Commit Graph
2554 Commits
Author SHA1 Message Date
Nicky LeachandClaude Fable 5.2 82ce8e64ba feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Core reads Runner events through the Codex app-server driver, also
for ACPX agents. The driver refuses a turn that the controller did not
start, except a goal turn of Codex.
> - This pull request lets the driver accept a turn that the provider
started, only directly after a turn that the Runner marked as waiting.
> - The benefit is that the next parts can use that turn. Nothing
reaches this code until the last part sends the opt-in.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 1. Its base branch is
`feat/adopt-claude-provider-started-turn`. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. What the driver does with a turn that the
provider started.

**Subsystem affected**

Paperclip Runner package, the controller side: `src/drivers/codex` (the
driver that Core uses to read a native session).

**Current behavior**

- A `turn/started` that the controller did not ask for fails the session
with `turn_binding_mismatch`, except for an active Codex goal.
- An interrupt with no active turn fails with "already terminal".

**Proposed behavior**

- A turn terminal that the Runner marked with `awaitingProviderTurn` and
that has no result puts the driver in a waiting state. The driver event
`turn.completed` carries the marker.
- In that state, a `turn/started` with `origin: "provider"` is admitted
the same way as a goal turn, and the driver event carries `origin:
"provider"`. In each other state it still fails closed.
- In that state with no active turn, an interrupt gives an
acknowledgement item with the status `provider_turn_wait_ended`. It
sends nothing to the Runner. The driver also gives that acknowledgement
when it gets the internal `paperclip/runTerminal` notification with a
private marker; a notification without the marker does nothing.
- An interrupt for a provider-started turn that is running is sent to
the Runner only when it names that turn.
- The driver saves the waiting state in the persisted session
(`awaitingProviderTurn`, written only during a wait). After a controller
restart, the restored session still admits the provider-started turn and
still acknowledges an interrupt during the wait. The driver restores the
state only when the session has no active turn, no stored result and at
least one finished turn, so a new run cannot take the wait of the run
before it.
- When a restart finds that the saved active turn is finished, the turn
read of the transport and `reconcile()` of the driver carry the same
marker as the live `turn/completed` notification.
`runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from
the saved `turn.completed` event to the turn object of a read.
- The driver ends the waiting state when the Runner reports the run
terminal.

**Reason and benefit**

The consumer in the next part needs these three signals: a marked
terminal, an admitted turn, and a clear end of the wait.

## What Changed

- `codex-session-terminal.ts`, `codex-session-notifications.ts`,
`codex-session-state.ts`, `codex-harness-session.ts`,
`codex-driver-values.ts`: 37 lines added, 4 removed.
- `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines).
- `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`,
`backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`:
the optional field in the two persisted shapes and its copy in each
direction.
- `live/runnerd-codex-transport.ts`: the marker on a recovered turn
terminal. The loop that builds the earlier terminals of a `thread/read`
is now the function `runnerdPriorTurnTerminals`, with the same logic, so
that a test can call it.

## State saved by the previous release

- One new optional field in the persisted driver session:
`awaitingProviderTurn`. A snapshot outside a wait has the shape of the
previous release.
- **State from the previous release, read by this release:** the field
is absent, so the driver does not wait. This is the behavior of today.
- **State from this release, read by the previous release:** the
previous reader copies the fields that it knows and ignores this one. No
reader rejects it, and the integrity digest covers the stored object as
written.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- 6 driver tests: a provider-started turn after a marked terminal is
admitted and can report a result; a provider-started turn with no marked
terminal fails closed; an interrupt during the wait is acknowledged; a
running provider-started turn is not interrupted by the end of the wait;
the marked notification ends the wait and a notification with no marker
does not; a normal terminal is not changed.
- Restart tests: a session restored from a snapshot with the waiting
state admits the provider-started turn and settles its result; a
snapshot outside a wait has no new field; a restored session with no
waiting state refuses a provider-started turn as today; an interrupt
during a restored wait is acknowledged. A saved active turn that the
read reports finished with the marker gives a `turn.completed` with
`awaitingProviderTurn: true`; with no marker it is an ordinary terminal.
- `harness-driver-backend.test.ts`: the field goes through a serialized
checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324
passed): the marker is on a recovered terminal only for a
`turn.completed` with a boolean `true`.
- `src/native-session-runtime.test.ts`: 297 passed, with the real
`paperclip-runnerd`.
- Mutation checks for the restart work, each made a named test fail: no
restore of the state; no copy in the backend; no copy in the transport;
no marker argument in `reconcile()`; no end of the wait at the run
terminal.
- Mutation checks: the acknowledgement branch removed, the admission
removed, and the marker check removed each made a test fail.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** The Runner sends the marker only after the
opt-in of part 4.
- **The admission rule is shared with Codex goal turns.** The existing
goal tests pass with no change.
- **Known limit after a controller restart.** If the Runner already runs
a provider-started turn and the saved active turn is the turn before it,
recovery fails closed, as today for each unexpected active turn.
- **A controller turn that starts during a wait** clears the waiting
state when its `turn/started` arrives, not when it is sent. A snapshot
in that short time can hold an active turn and the field together; the
restore rule drops the field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 20:45:49 -08:00
DottaandClaude Opus 5.5 9824277f65 perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Most runner events that are not text deltas
(`tool.execution.started/progressed/completed`, `item.completed`) wake
the chat progress projection
> - That wake is registered inside the append's transaction, so each
such event takes 4 statements (BEGIN, the run lock read, the append,
COMMIT), where an event without a wake takes 1 (#21511)
> - The chat progress projection rebuilds itself from the stored events;
the wake only tells it when to look
> - This pull request writes an event whose only wake is chat progress
with the one-statement append, and sends the wake after that statement
commits
> - The benefit is 1 statement instead of 4 for most non-delta runner
events, with response, reassessment, retention and recovery wakes
unchanged

## Linked Issues or Issue Description

Refs #21511 (one-statement append). Refs #21528 (lock read returns the
transaction id). Refs #21549 (load model at recorded rates).

**What existing behavior does this improve?**
`appendHeartbeatRunEvent` for native source events whose only wake is
chat progress.

**Subsystem affected**
Server run event ingest (`heartbeat-run-events.ts`), the chat progress
notification helper, and two additive exports in
`packages/db/src/work-signals.ts`.

**Current behavior**
A progress event opens a transaction, reads the run lock with
`runtime_mode`, the bound task and the transaction id, probes for a
replay, registers the chat-progress wake as a tracked intent, appends,
and commits.

**Proposed behavior**
When chat progress is the event's only wake, the append takes the
one-statement path of #21511. That statement also returns the run's
`runtime_mode` and bound task id from the row it locks, and, where
commit notifications are on, calls `pg_notify` inside the same statement
for runs that are native and bound to a task, so other server processes
get the wake at commit. After the statement returns, this process
publishes the wake as settled to its own listeners. A replay, a gap, a
changed run or any other fallback takes the existing transaction, which
registers the wake as before.

**Reason and benefit**
At recorded rates these events are about 61% of the non-delta events
(#21549), so this cuts about 3 statements per such event.

**Breaking changes**
None. No schema change.

## What Changed

- `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other
wake clear) also qualifies for the one-statement path.
- `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement
append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`,
rendered once) from the locked row, plus a conditional `pg_notify`.
- `chat-native-progress-notifications.ts`:
`chatNativeProgressCommitNotification` and
`publishChatNativeProgressSettled`.
- `work-signals.ts` (additive): `databaseWorkCommitNotification(owner,
topic)` returns the cross-process payload when notifications are on;
`publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled
wake to this process's listeners.

## Verification

- Red/green on embedded PostgreSQL. Each new test failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "appends an event whose
only wake is chat progress in one statement and wakes progress after it
commits": 1 statement (before: a transaction), the wake is published
once as settled with no intent, and a replay publishes nothing.
- "keeps the transaction for the last events of a turn and a run, and
one statement for progress-only events":
`tool.execution.started/progressed/completed` and a non-final
`item.completed` take one statement; a final `item.completed`,
`run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled`
and `turn.interrupted` keep the transaction.
- "sends the progress wake to other server processes with the one
statement's commit": with commit notifications on, another connection
that listens on the work channel receives exactly one chat-progress
notification, and a replay sends none.
- `chat-native-progress-work.test.ts` "wakes the projection from a
one-statement progress event after the statement commits": the real chat
progress worker publishes the progress for an event appended in one
statement.
- Mutation check: without the settled publish after the statement, that
test fails.
- Lost-wake repair (condition b): an event row stored without any wake
models a lost hint.
- "repairs a lost progress wake with the run's next progress event":
nothing is published for the stored event; the run's next progress event
publishes the progress.
- "repairs a lost progress wake with the startup reconcile": a worker
that starts after the stored event publishes it.
- The existing response-wake test now uses a final reply and still shows
the intent before the write, and the race test of #21528 uses a final
reply so it still covers the transaction path.
- Other focused suites, one file at a time: the port suite with
conformance (105), `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51), `work-signals` (25), atomic-append (31),
`chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run
typecheck` and `pnpm check:module-boundaries` pass.

- CPU per operation (a local script, not committed: 800
`tool.execution.progressed` appends to one native task run on embedded
PostgreSQL, one connection, three rounds, `origin/master` code against
this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms
after in the warm rounds; wall time per event 0.84–1.04 ms before and
0.52–0.53 ms after.
- Inside a caller's transaction: new test "keeps a progress-only event
inside a caller's transaction on the locked path, and settles its wake
only when that transaction ends". `publishDatabaseWorkSettled` publishes
at once only outside a transaction; inside one it waits for the
outermost commit and drops the wake on rollback (new
`work-signals.test.ts` test, which failed with the unguarded publish).
The one-statement path is only taken when the owner is not a transaction
(`!is(db, PgTransaction)`, #21511), so the two guards agree.

## Risks

- **Lost wake.** If the statement's outcome is unknown (the connection
drops after the server ran it), this process sends no wake for that
event. In a chat channel (Slack, Telegram) the progress line for that
task can then stay one step behind (for example it does not move from
"working" to the latest tool step) until the run's next runner event, a
run milestone, or a server restart. A turn normally ends with an event
that keeps the transaction
(`turn.completed/failed/cancelled/interrupted`), and the run's
completion notifies chat run milestones, which include chat progress
(`notifyChatRunMilestones` in `run-completion.ts` and
`native-run-finalizer.ts`). So progress cannot stay stale after a run
ends; in the worst case it lags one step for the rest of a turn. The
projection rebuilds from stored events, so nothing is lost or
duplicated.
- The chat-progress wake no longer publishes an intent before the write.
Intents let the coordinator fence a pass against an in-flight write; the
settled wake after the commit makes the projection run again after the
row is visible.
- `packages/db/src/work-signals.ts` gets two additive exports and no
change to existing behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:37:35 -06:00
Nicky LeachandClaude Fable 5.2 f7b1651799 feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The parts before this one give the adapter an ordered stream of
those cycles (#21421), declare the contract (#21422), and make
`paperclip-runnerd` wait for such a turn (#21423).
> - This pull request is the last part in the Runner. The sidecar reads
the stream, and gives a cycle to `paperclip-runnerd` as a
provider-started turn only when `paperclip-runnerd` armed it.
> - The benefit is that a later change in Core can let a run continue
when background work finishes. Until Core sends the opt-in, a production
run does the same as today.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request was first opened with the full change (49 files). It
is now split. The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. This pull request: the sidecar adopts a cycle.

Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code
at the head of this branch is the same as before the split; only the
base changed. Review comments made before the split are about code that
is now in the parts above. Refs #19912 (design).

**What existing behavior does this improve?**

What the ACPX sidecar does with a cycle that Claude starts by itself.

**Subsystem affected**

Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only.

**Current behavior**

- The sidecar does not read the cycle stream of the adapter (#21421), so
each cycle is declined and the background state becomes unknown.
- The sidecar does not know the command `session.provider_turn`, so
`paperclip-runnerd` (#21423) never waits.

**Proposed behavior**

- The command `session.provider_turn` with `armed: true` arms the
sidecar for one turn. The sidecar accepts it only for the terminal that
reported a running background shell, with no active turn and a provider
that is alive. One flag holds the arming. It is cleared at each turn
start and turn end, on suspend and close, and when the provider is lost.
- One loop, `consumeProviderCycles`, reads the stream. For each cycle:
- if the sidecar is armed and no turn is active, the sidecar adopts the
cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`,
then the events, the usage and the terminal of the cycle, as for a
normal turn;
- if the cycle cannot be decided yet (the prompt turn is still settling,
or `paperclip-runnerd` did not arm yet), the sidecar holds it for a
maximum of 5 seconds, and then adopts or drops it;
- in each other case the sidecar drops the cycle. The stream ends, and
the background state becomes unknown, so `paperclip-runnerd` restarts
the provider at the end of the turn as today.
- A Paperclip tool call or a permission request of a held cycle waits
for the decision of its own cycle. It is never bound to the prompt turn
or to a different cycle.
- If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar
publishes the same `plan.proposed` event as for a controller turn, and
the turn ends with the same terminal: completed, with no
`backgroundTasks`, so `paperclip-runnerd` does not wait again. A
controller turn is not changed.
- If the provider exits while the sidecar is armed with no turn active,
the sidecar sends a `runtime.diagnostic` with the code
`ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait.
- `turn.cancel` for a provider-started turn cancels the cycle and closes
the provider.
- With no arming, nothing is shown. No controller sends the opt-in yet,
so `paperclip-runnerd` never arms.

**Reason and benefit**

- Only `paperclip-runnerd` decides that a cycle becomes a turn. The
sidecar cannot start a turn by itself.
- One reader of one ordered stream means that two cycles cannot overlap.

## What Changed

- `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the
arming flag, `consumeProviderCycles`, `adoptProviderCycle`,
`pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is
not decided.
- `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines)
and two fake hosts in `src/test-support` (227 lines). One host follows a
script. The second runs the real cycle stream and the real background
task tracker.
- `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter
for a test harness.

No schema, no Rust and no persisted state changes in this part.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `pnpm run typecheck:typescript`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript`: no errors. The
server check `node scripts/verify-runner-vendor-dependencies.mjs`, the
cache guard and the contract tests: passed.
- `npx vitest run src/cli src/drivers/acpx
src/live/runnerd-acpx-provider-turn-wait.test.ts
src/drivers/runner-tool-bridge.test.ts
src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)`
files, `2147 passed`, 9 skipped.
- `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites:
all passed. No Rust changed in this part.
- The sidecar tests of this part: a cycle that starts before the arming
command, or while the prompt turn settles, is held and then adopted with
its tool call; a cycle is never shown when `paperclip-runnerd` did not
arm, disarmed, armed for a different terminal, armed for a quiet
terminal, or armed after the provider exited; two cycles in sequence
with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a
lost provider; Codex gets no change.
- Mutation checks, each made the named tests fail: the sidecar arms with
no matching terminal; no hold; the old hold condition that drops a cycle
while the prompt turn settles; no wait of a tool call for the decision.
- Live run with a real Claude account, not in CI, with the real ACP
runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the
adapter level: one background shell gave `{ live: 1 }`, one cycle with
its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{
live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were
admitted, and none was refused.

Not run: a live run through the real sidecar process,
`paperclip-runnerd` and Core (that path needs an API key, not a local
sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in, so the sidecar
is never armed. The new code that runs today is the loop that reads the
stream: with no arming it drops each cycle, and the background state
becomes unknown as before.
- **A follow-up for a command that finishes while Claude reports on an
earlier one is lost.** The Runner waits again only while a background
command is still running. The run ends as it does today. The reason is
in #21421.
- **A cycle that starts in the last second of a prompt turn** can have
its first messages in the prompt turn and the rest in the
provider-started turn. Tool calls stay safe: the admission rule of
#21278 refuses a call that the active turn did not announce.
- **The 5 second hold** is not tested under load. A cycle that is held
longer is dropped, and the run takes the path of today.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 19:50:22 -08:00
Nicky LeachandClaude Opus 5.5 4a78124a3e feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn
that ended with no result while a background shell is running, this
stops the shell and the run fails.
> - #21278 binds each tool call to its turn, and #21293 reports how many
background shells are running at the end of a turn. Part 4 declared the
contract.
> - This pull request makes `paperclip-runnerd` wait for a turn that
Claude starts by itself, when the controller asks for it, and admit that
turn into the same run.
> - The benefit is that the Runner has one place that decides to wait
and one place that ends a wait.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 5. Its base branch is `feat/prp-provider-turn-contract`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

What `paperclip-runnerd` does after a Claude turn ends with no result
while a background shell is running. Nothing changes until a controller
sends the opt-in.

**Subsystem affected**

Paperclip Runner: the ACPX provider backend in `runner-core` (Rust).
Claude only.

**Current behavior**

- After each completed turn the Runner emits `run.terminal`. A turn with
a running background shell gets the provider restart, and the shell
stops.

**Proposed behavior**

Nothing changes unless the controller sends `providerStartedTurns: true`
in the payload of `turn.start`. No controller sends it yet.

With that field:

- The Runner keeps the provider and waits when all of these are true:
the agent is Claude; the turn completed; the run has no accepted
`paperclip_finish` or `paperclip_block` result; the terminal event
reported `backgroundTasks` with `live` above zero; each condition of
#21293 for a kept process is true; the sidecar accepted
`session.provider_turn` with `armed: true`.
- `turn.completed` then has `awaitingProviderTurn: true`, and no
`run.terminal` follows. The Runner holds that `run.terminal`.
- When the sidecar reports a turn that the provider started, the Runner
admits it only during a wait and publishes `turn.started` with `origin:
"provider"`. In each other case that report fails the session closed.
- When a wait ends with no turn active, the Runner publishes the held
`run.terminal`. This applies when the provider process exits, the
sidecar exits, the controller cancels or stops the run, the session is
closed or suspended, and after a Runner restart.
- The held `run.terminal` and the removal of the wait are saved in one
write, and the controller gets the event from the next poll. A crash
after a cancel or a stop cannot lose it, and it is never sent two times.
- A controller `turn.start` during a wait replaces the provider first,
as before, and the held event is dropped.
- A `run.attach` from a different run during a wait ends the wait when
the attach succeeds. The Runner then discards the held `run.terminal`,
because that event is of the old run and must not end the new run. This
is the same rule that a Runner restart applies. The wait is removed only
in the state that the attach saves for the new run. If the attach fails,
for example when the old session does not stop, the old run keeps the
wait and gets its held `run.terminal` one time, from the next poll or
from a cancel or a stop. A `run.attach` from the same run publishes the
held event to that run.
- The Runner waits at most 5 seconds for the sidecar to answer a request
to arm or to disarm a provider-started turn. This is the limit that turn
cancellation already uses. Before, the limit was the 30 seconds of an
ordinary request, and a cancel, a stop or a shutdown had to wait behind
it. No answer is handled as a lost sidecar: the session fails closed,
and the Runner still publishes the held `run.terminal` one time.
- The sidecar of this branch does not know `session.provider_turn`, so
the arming request fails and the Runner does not wait. The behavior
comes with part 6.

**Reason and benefit**

- The wait is one entry in the saved session state. One rule ends it at
each save, so each new way to leave the open session ends the wait with
no new code.
- A wait must not outlive the provider or the sidecar, and a cancel must
stop it.

## What Changed

- `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests):
`provider_turn_wait { turn_id, held_run_terminal }` in the saved state;
the rule that ends a wait at save; `settle_turn_events`, which decides
at one place to arm, to create the wait and to put
`awaitingProviderTurn` on `turn.completed`.
- `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its
terminal, the arming request, and `admit_provider_turn`.
- `acpx_provider_session.rs`, `acpx_provider_backend.rs`,
`acpx_provider_state.rs`, `provider_events.rs`,
`acpx_park_preparation.rs`: the opt-in from `turn.start`; the record
that the run has a result; the provider-started turn in the state and in
the PRP events; cancel and stop with no turn; no turn-end restart and no
park during a wait.
- `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no
event also checks that the sidecar process is alive. A child process can
keep the output stream of a dead sidecar open.
- `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and
failures for tests.
- Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`,
`tests/support/acpx_quiet_turn_restart.rs`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon),
`src/native-session-runtime.provider-turn.test.ts` (the controller of
today).
- `runner/DURABLE_TRANSPORT.md`: the rules.

## State saved by the previous release

- One new optional field in the saved ACPX session state:
`providerTurnWait`. It is written only during a wait. In each other
state the saved file has the shape of the previous release.
- **State from the previous release, read by this release:** the field
is absent, so there is no wait. A unit test covers this.
- **State from this release, read by the previous release (a
rollback):** outside a wait, the file loads as before. During a wait,
the previous Runner refuses the file with its "ACPX provider state is
malformed" error, because its loader refuses each unknown field. Only a
run that waits at the moment of a rollback is in this case, and no run
waits before a controller sends the opt-in.
- **Runner restart during a wait:** the wait is restored. The provider
is a new process, so the wait is over, and the controller receives the
held `run.terminal` one time. A test with the real daemon covers this.
- **Controller of today or of the previous release:** `turn.start` has
no opt-in. A test runs a turn with a running shell and a turn with an
unknown background state, and compares the events, the commands after
the turn, the restart, and the park state before and after the restart.
They are the same.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored.
All 17 `acpx_*` suites passed.
- `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the
real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends
with exactly one `run.terminal` when the sidecar exits while a child
process keeps its output open, when the provider is lost, when the
controller cancels, and after a Runner restart; with no opt-in,
`run.terminal` follows the turn and the sidecar is never armed.
- Rust tests: wait and admit; no wait with a result; no wait with no
opt-in; an active goal neither marks nor arms a wait; a sidecar that
refuses to arm; a provider-started turn that the Runner did not wait for
fails the session closed; cancel and stop during a wait; provider and
sidecar loss; restart; the saved shape in both directions.
- `src/native-session-runtime.provider-turn.test.ts`: the real
`consumeTurn` of today gets the event sequence with no opt-in and
finalizes it as `native_finalization_missing`, as today.
- Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a
different run attaches after the snapshot reports `warmAttachReady:
true`; the new run gets no `run.terminal`, the saved state has no wait,
and the old session is closed. A same-run attach gives exactly one
`run.terminal`. An attach that fails when the old sidecar is dead leaves
the wait with the old run, and the old run then gets exactly one
`run.terminal` from a poll, `run.cancel`, `turn.interrupt` or
`turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in
the pending events of the new run; with the wait ended before the attach
is complete, the old run gets no `run.terminal` after a failed attach.
- No answer from the sidecar, in
`tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not
answer the disarm request; `run.cancel` and `turn.interrupt` each return
`already_settled` in less than 10 seconds, exactly one `run.terminal`
reaches the run, `session.close` completes in less than 2 seconds with
no second `run.terminal`, and the provider process is gone. With no
answer to the arm request, the turn ends with its `run.terminal` and
with no wait. With the limit removed, each test fails at 30 seconds.
- Mutation checks, each made the named tests fail: the rule that ends a
wait at save; no `run.terminal` after a restart; arming with no opt-in;
the marker set with no wait; the sidecar process check removed.

Not run: a rollback with the binary of the previous release; a run with
the opt-in and a sidecar that answers the arming request with an error
(the code reads each error as not armed); Linux; the full suite. CI
supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in.
- **A rollback of the Runner during a wait** makes the previous Runner
refuse that saved session. See "State saved by the previous release".
- **The wait has no time limit in the Runner.** The controller must
close, suspend or cancel. Core will own that limit.
- **Park is refused during a wait.** The controller must end the wait
before it parks.
- **A new run can attach during a wait.** The attach is not blocked,
because the Runner has no time limit on the wait and a new run can send
only `run.attach`. After an attach that succeeds, the old run gets no
`run.terminal` from the Runner; the controller must not depend on that
event. The request that disarms a sidecar that the attach keeps has no
test, because no test can keep a session across an attach during a wait.
- **After a sidecar that does not answer, `runner.suspend` returns an
error at once** ("provider lifetime cleanup is not yet proven"), and
`session.close` completes. This is the behavior of today for each lost
sidecar; this PR does not change it.
- **The shared code that changed** runs for each Claude turn: the record
of a run result, and the settle step at each turn end.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:21:10 -08:00
DottaandClaude Fable 5.2 689f3b190e fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic recovery continues interrupted and failed tasks without a
person, with a per-task limit of 6 automatic attempts per hour
> - There is no company-level limit. When the sandbox provider refuses
to create sandboxes, every task fails the same way, and recovery keeps
creating new runs for each task
> - On a hosted instance, one hour of a provider outage made 510
automatic runs, 587 failures and 0 successes across 289 tasks. No task
had more than 4 runs, so the per-task limit did not engage. Each start
asked a refusing provider for a sandbox and used server memory, which
made the outage longer
> - Recovery must still continue every task by itself. The answer is to
wait while the infrastructure fails, not to give the task to a person
> - This pull request adds a company-scoped pause for automatic work: it
opens when recent automatic runs fail before the agent starts, the
recovery passes stop creating new automatic runs while it is open, and
it closes by itself when runs start normally again
> - The benefit is that an infrastructure outage no longer multiplies
itself through recovery, and recovery resumes with no person acting

## Linked Issues or Issue Description

**What happened?**
During a sandbox provider outage, automatic recovery created hundreds of
new runs per hour across hundreds of tasks. All of them failed before
the agent did any work. The per-task limit did not engage because no
single task reached it.

**Expected behavior**
When most recent automatic runs of a company fail before they start,
recovery waits instead of creating more runs. It tries again at a small
rate. It resumes at full rate when runs start normally again. A person's
work is never held.

**Steps to reproduce**
1. Make the sandbox provider refuse lease creation (HTTP 503) for a
company with many assigned tasks.
2. Let recovery continue the failed tasks.
3. Count automatic runs created per hour: it grows with the number of
tasks, and every run fails at sandbox creation.

Refs #21394 (continuation classification this change will reuse when it
lands), #21433 (start-pass rewrite that the start hook will build on).

## What Changed

- New table `automatic_run_brakes` (one row per company, cascade delete
with the company; migration `0480`). Migration `0481` builds
`heartbeat_runs_company_finished_idx (company_id, finished_at desc)
where finished_at is not null` concurrently, so the signal reads the
runs that finished in the window directly. It stores the state (closed,
open, half_open), the reason with the window numbers, and the
compare-and-set fields that two servers share. A company with no row is
closed.
- New module `server/src/services/recovery/automatic-run-brake.ts`:
- Signal: on the database clock, the automatic runs of the company that
finished in the last 10 minutes (and since the last state change),
newest 500 by finish time, plus up to 200 running runs. A run counts as
a failure only when it ended in its own start phase: no provider work
(`executionRecovery.providerWorkStarted = false` or the native
`nativePreDispatchFailure` record), a legacy run still preparing, or a
transient setup code. A run interrupted by a server stop, restart or
lease loss is neutral, so a release swap does not open the pause. A run
that succeeded, or ended for a person-owned or task reason, counts as
healthy.
- Opens at 12 or more counted runs with 80% or more failures, or on 5
sandbox refusals (`environment_provider_unavailable`) in a row.
- Each outcome counts once, at its own time: a finished run at its
finish time, a running run that got past setup at its start time. Read
time never orders evidence.
- Closes in two steps: open moves to half_open after at least 2 minutes
when the newest 2 counted runs are healthy or the newest one succeeded;
half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64)
and then closes. It reopens from half_open if the signal trips again. It
closes by itself after 15 minutes with no automatic activity.
- Reads are one indexed query per company, cached for 5 seconds in a
bounded in-process map (256 companies). Each state change writes one
structured log line
(`automatic_run_brake.opened|half_open|closed|reopened`) with the
numbers.
- It also contains the start gate `holdAutomaticRunStart` (one probe at
a time, at most one new probe per 60 s, release budget per window by
compare-and-set, no I/O while the cached "held until" time has not
passed, person origin taken as an argument with a cached fallback). This
pull request does not call it from the start pass; see Risks.
- Recovery passes defer creating runs while the pause is open. The gate
is only where a run is created:
- `deliverReconciledExecutions`: after the agent-room check and right
before the wake, the delivery asks for a creation slot; when the pause
holds it, the continuation stays pending. The held backlog
(`continueHeldExecutions`) is not gated: recording a pending
continuation creates no run and costs nothing, and its delivery passes
this gate once.
- Assigned-work sweep `act()`: after the process rate limit and before
it claims a continuation, it asks for a creation slot; when held, it
returns `deferred_rate_limit` with the next try time, so nothing is
written to the per-task attempt ledger. New assignments with no earlier
run are not deferred.
- A reserved slot is given back (compare-and-set on its own timestamp)
when no run is created: the wake is refused or throws, the task is
capped, the claim changed, or the task went back to review. The
give-back is conditional in SQL on no run for that task having been
created since the reservation, so a wake that commits its run and then
throws keeps the slot.
- The sweep's overdue saved-work recovery path returns before the slot
is asked for: it starts a recovery attempt, not a run, and is never
held.
- Person work is never deferred. The gate is called only at the delivery
(`agentHasRoom`) and the sweep (`act()`), and each call passes the task
and the run it continues. While the pause is open, the gate does not
hold the record when any of these is true; it holds only when none is.
With no task or no source run it does not hold (fails open).

| Shape | How it is recognised |
| --- | --- |
| A Retry or "move forward" waiting for the task
(`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an
unstarted wake request from a user for the task |
| A Retry folded (`coalesced`) into the continuation, or a Retry while
the delivery is pending | a wake request from a user for the task,
requested after the source run finished (any status) |
| A person's message or a new assignment by a person after the last run
| the same: a wake request from a user for the task after the source run
finished |
| A held task selected because a person pressed Retry (the refused Retry
run) | a run for the task created after the source run finished whose
wake is a person reason (`task_retry`, chat message), or that has
`taskRetryRequestId` or `explicitUserContinuation` |
| A continuation of a person's Retry (any depth) | the source run's
chain reaches a person's run (`listPersonOriginSourceRunIds`, the
dispatch rule) |
| A Retry during pending delivery that becomes the continuation itself
(`deliverContinuationAsRun`) | not gated: that run is created by the
person's own wake, never by these passes |
| The held backlog's saved-input path | not gated: recording or
releasing a held continuation creates no run here |
- To prevent a deadlock, a pass may still create one automatic run per
minute per company when no automatic run of the company started in the
last 2 minutes, however many runs wait (compare-and-set, shared by
servers).
- `GET /api/companies/:id/assigned-work-invariant` now returns
`automaticWork` with the state, the time it paused, the window numbers
and a plain sentence, for example "Automatic work is paused for this
company since T because 47 of 50 recent automatic runs failed before
starting."
- `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is
now exported, unchanged, so the gate uses the same person-origin rule as
dispatch.
- The outcome rules (person-owned codes, restart codes, transient setup
codes, native pre-dispatch record) come from their owning modules
through one memoised dynamic import. A static import of
`heartbeat/retries.ts` made a module cycle at server start.

### What the pause consumes, and how waste is bounded

| Token | Taken | Given back or bounded |
| --- | --- | --- |
| Creation slot (one per minute per company while open) | right before
the run is created, after every other check | released only when no run
for the task was committed after the reservation |
| Probe slot (start gate) | when the start gate lets one automatic run
through | another run may take it when the reserved run has not started
within 10 seconds (claim declined, agent paused or full) or was
cancelled, deleted or finished |
| Gradual-release start (start gate) | one per run the start gate lets
through | a declined claim spends one start of that window; the window
ends within 60 seconds and the next one doubles |
| Probe success count | once per probe run, by compare-and-set on its id
| a probe run counts once; cancelled or deleted probes count as neither
success nor failure |

### When the pause opens, stays open and closes

| State | Leaves when | Bound |
| --- | --- | --- |
| closed | opens when, in the last 10 minutes since the last change, 12
or more counted automatic runs finished and 80% or more failed before
starting, or the newest 5 counted runs were sandbox refusals | none
needed |
| open | moves to half_open when it has been open 2 minutes or more and
the newest 2 counted runs are healthy, or the newest one succeeded |
evidence arrives at least once a minute: when no automatic run started
in the last 2 minutes, a pass may create one per minute |
| open | closes when no automatic run started or finished for 15 minutes
(queued runs do not count) | 15 minutes after the last activity |
| open | stays open only while new automatic runs keep failing before
starting | each new try is spaced by the existing passes and by one
creation per minute when nothing starts |
| half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6
minutes |
| half_open | reopens when the signal trips again, counted only since
half_open began | none needed |

Restart-class endings (server stop, restart, lease loss, process loss,
interrupted native session) are neutral in every state, so a release
swap does not open the pause.

## Verification

- `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests,
pass): classification; opens on 20 failures before start; 50 queued
automatic runs give exactly one probe and no failure or attempt; a
person's Retry, a new assignment, a run a person joined and the
continuation of a person's Retry are not held; mixed and person-owned
outcomes stay closed; old data is ignored; 40 working runs interrupted
by a restart stay closed, then the same continuations failing at sandbox
acquisition open it; 5 refusals in a row open it; failed probe waits for
the interval, two probes past setup move to half_open, the release gives
exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen
from half_open; probe slot frees after cancel, delete or no start; quiet
close after 15 minutes; two instances on one database agree and start
one probe; one creation per minute while open across servers; 250 held
runs over 100 passes use at most 5 queries (at most 255 with the
person-origin fallback, one per run, once); caches stay at 256 companies
and 1024 runs.
- `npx vitest run
src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one
test per person shape above at the delivery and the sweep, with the
pause open and no creation slot, plus a control that still holds a
continuation no person asked for): includes "no pause record: every pass
behaves as before", "the waiting run cannot start: a pass continues a
task on another agent, then the pause closes", "one held task spends one
creation slot", and "a person's Retry or message is never paused at the
delivery, the sweep or the held backlog".: sweep, delivery and held
backlog create nothing while open, write no attempts, and continue by
themselves after recovery; the sweep still continues one task a minute
when nothing automatic waits. With the three deferral lines removed, all
4 fail (red), and they pass with them (green).
- The pause reads as closed and logs once when its run rules cannot load
(test in `automatic-run-brake.test.ts`).
- `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the
module loads after the heartbeat service, in the server's import order.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify`: see the comment below for the result. CI supplies the
full suite.

## Risks

- This pull request does not hold starts. Already queued runs and
`transient_failure_retry` runs still start while the pause is open; they
act as the natural tries. The start gate goes in as a follow-up on top
of #21433, at a point before that pass counts a claim attempt and reads
the full run row, so a held run costs no attempt and no read.
- The sweep reuses the `deferred_rate_limit` outcome for a deferral, to
keep its outcome type unchanged while #21394 edits that file.
- The automatic-run predicate is built from master exports
(`RECOVERY_WAKE_REASONS`, retry markers). It will switch to
`isAutomaticContinuationRun` when #21394 lands.
- Migration `0480` adds one small table.
- Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`)
runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement:
it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in
`packages/db/src/client.ts`, like 0470 and 0476, and the runner drops
and rebuilds an invalid index after an interrupted build. The index is
declared in the Drizzle schema, so `pnpm db:generate` reports no
changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On
a large `heartbeat_runs` table the build is online (no write lock) but
takes time proportional to the table. After it exists, it is one more
index on a hot table: each run that finishes writes one more index
entry.
- No user-visible text changes outside the operator report.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell,
file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:20:09 -06:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
DottaandClaude Opus 5.5 a726dce4f0 feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools from connected apps through the Paperclip MCP
gateway
> - Some tool calls take a long time: exports, builds, large searches
> - The MCP progress utility lets an app report how far a call is, but
Paperclip ignored it
> - So the task chat showed only a spinner until the call finished, with
no sign of life
> - This pull request reads the app's progress and shows it on the
tool's row while the call runs
> - The benefit is that people can see that a long call is working, how
far it is, and how long it has run

## Linked Issues or Issue Description

**Subsystem affected**
Connected apps (MCP gateway) and the task chat.

**Problem or motivation**
A long tool call from a connected MCP app shows only a spinner in the
task chat. Paperclip does not send a `progressToken`, and it drops
`notifications/progress` from the app.

**Proposed solution**
Send a `progressToken` on each `tools/call` that an agent run makes.
Read the app's progress notifications from the call's response stream.
Show a progress bar (or an indeterminate sweep), the latest message, and
elapsed time on the running tool row. Keep the latest state so that a
reload mid-call shows it.

**Alternatives considered**
A new transport for progress. The existing company live-events socket
already carries per-run events with run read checks, so this change uses
it.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress

## What Changed

- `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`.
`readMcpHttpResponse` gives each notification on the response stream to
an observer before the result. The observer cannot fail or delay the
call. New `mcpProgressNotificationOf` matches a notification to its
token.
- `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an
agent run send a per-call token and forward progress to a reporter.
Calls without a run send no token.
- New `server/src/services/tool-call-progress.ts`: checks each update
(finite, increasing, plain single-line text of at most 200 characters).
It publishes a `heartbeat.run.tool_progress` live event at most every
250 ms (latest wins) and saves the latest update on the invocation at
most every 1 s. Nothing is awaited by the call. A final `done` event
goes out when the call ends.
- DB: nullable `tool_invocations.progress` jsonb column (migration
`0479`, `ADD COLUMN` only). A save never writes onto a finished call.
- Live delivery: the new event follows its run like transcript output. A
client gets it only for runs it follows, after the run read check, in
the run's company only.
- New route `GET /api/issues/:issueId/tool-call-progress` (task read
check) returns the latest progress of the task's running calls, for a
reload mid-call. Documented in `docs/api/issues.md`.
- UI: `ToolCallProgressProvider` (one per chat) keeps the latest update
per call. A running tool row subscribes and shows a bar or sweep, the
message, the percent and elapsed time (design tokens, reduced motion
honored). Finished rows drop it and never fetch or re-render for
progress. `LiveUpdatesProvider` ignores the event for query
invalidation.
- `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool
(`steps`, `stepMs`, `withTotal`) in its own file.
- Storybook: `Tasks/Tool progress` stories (running with total, running
without total, done).
- Docs: new `doc/connections/MCP-PROGRESS.md`;
`MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported.

## Verification

- Red/green: on the original source, 5 of the new server tests fail
(token in `_meta`, SSE with interleaved progress and result,
notification matching, run-interest delivery, gateway end-to-end). With
this change they pass.
- `server/src/__tests__/mcp-http.test.ts` (13),
`tool-call-progress.test.ts` (9: throttling, latest wins, bounded text,
out-of-order drop, finish, failures never throw, company/run/tool
scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres:
company and task scoping, no write onto a finished call),
`realtime/live-event-delivery.test.ts` (run interest),
`tool-gateway.test.ts` end-to-end against the real SDK kv demo server
over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7
throttled events, all for the right company, run, task and tool, the
last one `done`; no event reaches another company; the invocation keeps
the progress; the result is unchanged.
- UI: `tool-call-progress.test.tsx` (10: name matching, row states,
plain-text rendering, drop on complete, restore after reload, no fetch
for finished rows). All 67 task-chat test files pass.
- `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP
with and without a total.
- Storybook: checked the three stories in a browser. The bar fills to
40%, the sweep animates, the elapsed time ticks, and the done row shows
no progress.
- `pnpm verify`: see the CI checks for the final revision.

## Risks

- Low. The migration adds one nullable column.
- An SSE response is now bounded per message, not per stream, so handled
notifications (such as progress) do not use up the result's size limit.
Each message is still bounded.
- Progress code runs only while a run's tool call is in flight. It adds
a few bytes to the request and throttled, fire-and-forget work. Agent
boot and time to first token do not change.
- A row finds its call by run and tool name. If one run calls the same
tool twice at the same time, both rows show the newer call's progress.
- Progress from a call that became an MCP Task is not shown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:14:22 -06:00
DottaandClaude Opus 5.5 81d06cfc97 feat(mcp-tasks): add a demo MCP server that uses Tasks, test against it, and keep audit records out of agent errors (#21464)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools in connected apps through the tool gateway, which
speaks MCP to each app
> - MCP Tasks (#21263) lets an app answer a long tool call with a
durable task. It shipped behind the experimental setting
`enableMcpTasks`, off by default
> - No public app returns tasks without credentials, so nobody had seen
the feature work against a real server, and it was hard to understand
> - This pull request adds a small demo app that uses Tasks, tests the
gateway against it over HTTP, and records a live end-to-end run
> - The live run found one bug: every task stop sent the call's audit
record, with a server stack trace, to the agent. This pull request fixes
it
> - The benefit is a repeatable way to try and test Tasks,
plain-language docs, and evidence for the decision to turn the setting
on

## Linked Issues or Issue Description

Refs #21389
Refs #21263

## What Changed

- Add an MCP Tasks demo server to `packages/kv-demo-mcp-server` (new bin
`paperclip-tasks-demo-mcp-server`; run `node
packages/kv-demo-mcp-server/dist/tasks/main.js` after a build). It is
stateless MCP 2026-07-28 with `server/discover`, and it declares
`io.modelcontextprotocol/tasks`. It has six tools: `slow_report`,
`needs_approval` (form question), `will_fail`, `cancellable`, `expires`,
and `quick_echo`. It saves task state to a file, so it can restart
mid-task. It needs no credentials. It has a status page at `/` and JSON
at `/api/state`.
- I put it in the existing demo package, so there is no new workspace
package and no `pnpm-lock.yaml` change.
- Add `server/src/__tests__/mcp-tasks-demo-server.test.ts`. It runs the
real gateway against the real demo server over HTTP, with one test per
tool plus a restart.
- Fix: `server/src/routes/tool-gateway.ts` no longer sends
`details.execution` to callers. That record has request IDs and the
server-side error stack. The audit still keeps it. Add
`server/src/__tests__/tool-gateway-agent-errors.test.ts` (red before the
fix, green after).
- Docs: `doc/connections/MCP-TASKS.md` now has "In plain words", "What a
person sees", "Try it yourself", the live-run results, and a
recommendation. The package README documents the demo server.
- `enableMcpTasks` stays off by default.

## Verification

- `packages/kv-demo-mcp-server`: `npx vitest run` gives 25/25 pass (13
new demo-server tests). `tsc -p tsconfig.test.json` is clean.
- `server`: `mcp-tasks-demo-server.test.ts` gives 8/8 pass.
`tool-gateway-agent-errors.test.ts` gives 2/2 pass, and it failed 2/2
before the fix.
- Existing suites that cover the changed route and Tasks:
`mcp-tasks.test.ts` and `tool-gateway-discovery-http.test.ts` (results
in a comment below).
- `pnpm verify` exits 2, because the doc is a global CI input. Required
CI gives the broad coverage.
- Live end-to-end run on a local `test-drive` instance with
`enableMcpTasks` on. The agent was Codex `gpt-6-luna` at low effort. The
demo was connected through "Connect your own MCP server":

| Behaviour | Result | `tools/call` count at the app |
| --- | --- | --- |
| Polling (`slow_report`, 45 s) | Delivered after 22 polls | 1 |
| Question (`needs_approval`) | The card was answered in the UI.
`tasks/update` was sent, and the task finished | 1 |
| Failure (`will_fail`) | `provider_task_failed` with the app's message
| 1 |
| Cancel (`cancellable`) | Stop on the run sent `tasks/cancel` | 1 |
| Expiry (`expires`, 15 s) | `provider_task_expired`, no hang | 1 |
| Restart (`slow_report`, 120 s) | The server was down for 2 min 23 s.
The result was delivered once, and the agent was woken once | 1 |

After the run, all 6 `tool_mcp_tasks` rows were settled and none had a
next poll.

## Risks

- Low risk. The demo server is new code in a demo package, and the bin
is additive.
- The route change removes `execution` from tool-gateway error
responses. No UI code or test reads that field from these responses. The
audit and tool-call events still record it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:58 -06:00
DottaandClaude Opus 5.5 a4c570aede feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected MCP apps through the tool gateway.
> - An MCP app can ask the person a question in the middle of a tool
call (MCP elicitation). There are two kinds: a form, or a link to open.
> - Today a form question is a dead end:
> - A 2026-07-28 server's form `input_required` stops with "needs
information Paperclip can't collect yet".
> - An older server's `elicitation/create` makes a card, but the answer
is never sent back, and the server waits on an open stream.
> - Paperclip does not declare form support, so compliant servers never
ask.
> - This pull request shows the question as a card in the task chat. The
card names the app, and only a person can answer it. When the person
answers, the agent wakes. The agent's repeated call sends the answer to
the app exactly once, for both protocol eras.
> - The benefit: apps that need a choice or a detail from the person now
finish their work. Restarts lose nothing, and no call is left waiting.

## Linked Issues or Issue Description

Refs #21140 (stateless MCP client; this PR closes its "form elicitation"
follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs
#21389 (MCP Tasks follow-ups: the card answer validation item).

**Is your feature request related to a problem?**
When a connected MCP app asks the person a form question during a tool
call, the call stops and cannot continue. A 2026-07-28 server gets a
"needs input" error. An older server gets a card whose answer never
reaches it.

**Describe the solution you'd like**
Show the question in the task chat. Let a person answer, skip, or leave
it. Wake the agent when the question is resolved. Send the answer on the
agent's repeated call, once, with the server's exact state.

**Describe alternatives you've considered**
Holding the tool call open until the person answers. We rejected this:
it holds a stream and a run slot, and it does not survive deploys or
restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records
nine design decisions with their alternatives.

## What Changed

- **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec,
the inventory, the decisions, the journey, tests, and risks.
- **New module `server/src/services/mcp-elicitation.ts`.** It is the
single mapping for direct calls and MCP Tasks:
- It turns each spec form field into a real card field: text with
defaults, number, yes/no, single choice (`enum` and `oneOf`), and
multiple choice (`items.enum` and `items.anyOf`).
  - It turns card answers back into typed `content`.
- Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or
expired sends `cancel`.
- It refuses forms that ask for passwords, keys, tokens, card numbers,
or one-time codes.
  - It keeps app text plain, so links in it are not clickable.
- It seals `requestState` with AES-256-GCM, bound to the company and
connection.
- **New table `tool_mcp_elicitations`** (migration 0479). It is
company-scoped, with one row per question round. It holds a keyed call
identity, the sealed state, the questions and their cards, a claim time,
and a 24-hour expiry.
- **Gateway, stateless servers:**
  - Paperclip declares `elicitation: {form, url}` on task tool calls.
- A form-only `input_required` saves the round, makes the cards, and
stops with a wait message.
- A repeated call before the person answers stops again without
contacting the app.
- After the answer, the repeated call claims the round and sends
`inputResponses` and the exact `requestState`, once.
- A failed send releases the claim. A crash mid-send lets the claim
lapse.
- **Gateway, older servers:**
  - Paperclip declares form support at `initialize`.
  - The first question gets `cancel` at once, so the server never waits.
- When the same question (same words and schema) arrives on the repeated
call, Paperclip answers it from the card.
- **Gateway, mixed rounds and retries:**
  - A form beside a URL step shows the link first.
- A repeated call that stopped for a question runs again in the same
run. It is no longer replayed by the per-run idempotency key.
- **Cards:** the title is "*App* needs your input". The resolver policy
is `human_only`. "Send to *App*" sends the answer. A user comment does
not supersede the card.
- **Activity log:** new events `tool_gateway.elicitation_sent`,
`tool_gateway.elicitation_refused`, and
`tool_gateway.elicitation_expired`. They record the action only, never
the values.
- **MCP Tasks:** task cards use the same mapping and gain the app name,
real fields, and `human_only`. A secret or an unsupported form stays
unsupported.
- **Storybook:** `Tasks and conversations/Questions and approvals/MCP
App Questions` has five states: Asking, Answered, Skipped, Expired, and
TwoQuestions. The fixtures are generated from the server mapper's
output.
- **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the
README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are
updated.

## Verification

- `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass
on the branch.
- **Red/green:** 9 of the 9 gateway tests fail with master's
`tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new
module.
- The tests use a fake 2026-07-28 server and a fake 2025-06-18 server
that asks over the open stream and waits for the reply.
- Covered: answer, skip (`decline`), withdraw (`cancel`), two questions
at once, a restart while waiting, a crash mid-send, another company with
the same app and arguments, expiry, a refused password field, an agent
that tries to answer (rejected), and capabilities.
- Existing suites: `mcp-tasks`, `mcp-protocol-versions`,
`tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See
the results comment below.
- Storybook: the five stories render in headless Chromium, and the play
tests pass. The only console errors are the 404s from the backend-less
API calls that every story makes.
- `tsc --noEmit` is clean for `server` and `ui`. `node
scripts/storybook-catalog.mjs` passes.

## Risks

- **Older servers may start asking questions.** Declaring form support
at `initialize` can make older servers ask where they did not before.
Mitigation: the first question always gets `cancel` at once, so no call
hangs, and the agent gets a clear next step.
- **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one
indexed lookup for a waiting round. Nothing is added before the first
token or to agent boot.
- **Migration.** One additive table with no backfill, so it is safe to
apply online.
- **Changed Tasks behavior.** A task card that is withdrawn or closed
now sends `cancel` instead of `decline`, as the spec defines. A skipped
card still sends `decline`.
- **Stale cards.** Expiry is checked only when the agent calls again. A
card can sit past 24 hours until the next call. Then it is withdrawn and
the app asks again.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, file edits, headless browser), with extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:56 -06:00
Devin FoleyandPaperclip 0a96f972e1 Connect Tailscale networks through device sign-in (#21526)
## Thinking Path

> - Paperclip manages AI agents and the environments where they run.
> - SSH environments can reach machines through a company Tailscale
connection.
> - The initial connector requires a managed OAuth app that Tailscale
limits to its own tailnet.
> - Customers need to connect their own tailnets without registering an
app or supplying a key.
> - This pull request uses Tailscale's normal device sign-in for new
connections.
> - Customers approve the Paperclip device and then select their
machines in SSH environments.

## Linked Issues or Issue Description

**Problem or motivation**

The current Tailscale enrollment path cannot serve independent customer
tailnets through the documented self-service OAuth app model. It also
requires Cloud broker configuration and pilot admission before a
customer can start.

**Proposed solution**

Start a private Tailscale device in the tenant and open its normal
sign-in link. Save its encrypted device state before reporting a
successful connection. Retain the existing SSH machine selection and
agent account behavior.

**Alternatives considered**

Auth-key entry would require customers to copy a credential. A public
OAuth integration still needs a confirmed registration path. Device
sign-in uses a documented Tailscale flow without either requirement. See
[device
authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication)
and [OAuth app
limitations](https://tailscale.com/docs/features/oauth-apps).

**Related PR search**

Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for
`tailscale`. This follows the merged network connector in
[#20617](https://github.com/paperclipai/paperclip/pull/20617) and
preserves the SSH agent-account support in
[#21248](https://github.com/paperclipai/paperclip/pull/21248). The open
[#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes
storing an OAuth client and checking its health; this change instead
enrolls a device through interactive sign-in so customers do not need an
OAuth app.

## What Changed

- New connections use device sign-in. Existing connected devices and
pending legacy OAuth callbacks remain supported.
- Pending sign-ins are bound to their company and initiating user. They
have a fixed ten-minute deadline. Device state uses server-owned vault
secrets with trusted version and digest checks.
- The active Tailscale CLI waits for consent and device approval.
Cancellation, expiry, secret deletion, and stale completion cannot
restore a cancelled connection.
- The connector stays open during sign-in. It provides a link when
popups are blocked and shows device approval separately. Progress checks
stop on completion, failure, expiry, or unmount.
- Shared response types, OpenAPI contracts, SDK source pins, and the
API/setup documentation describe the new flow. No database migration or
Cloud provider configuration is required.

## Verification

- 109 focused tests cover network processes, the real database and
encrypted vault, routes, UI, and legacy connection behavior.
- Server and UI typechecks pass.
- Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure
locally. All 70 OpenAPI and Tailscale route tests and server typecheck
pass after synchronizing the API contract.
- Follow-up `79de05f38b9`: the transient-recovery regression failed
before the fix; all 20 connector UI tests and UI typecheck pass after
it. The query-error policy check reports no findings.
- The public SDK input registry check passes.
- [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2)
passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57
successful checks, no failures or pending checks. The interrupted
canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile
is 5/5 on this head with no unresolved review threads.
- `pnpm verify --explain` exits 2 because the documentation change
selects broad CI coverage. Focused local checks passed; broad CI passed
on `29cf296a597e470d432b84767050bc539caa280e`.
- Live local qualification on
`29cf296a597e470d432b84767050bc539caa280e`: native device sign-in
completed with interactive user approval. An isolated Ubuntu 24.04 ARM64
EC2 machine appeared in discovery and passed SSH pairing and workspace
probes. The assigned Claude Code agent used a Paperclip-managed
subscription, created `hello_world.txt` containing exactly `Hello,
world!` plus a newline, posted a completion comment, and marked the task
done. An independent read through the private connector verified the
bytes, remote user, and hostname. The remote machine had no stored
Claude login and no leftover temporary credential directories.
- Restarted the local Paperclip server and verified the same Tailscale
device identity, online peer discovery, successful SSH probe, and
unchanged file without new consent. Temporary public SSH ingress was
removed.
- Local qualification covers one user tailnet and the final Core
revision. Browser UI acceptance, two unrelated tailnets, a hosted
restart, and an agent task on the final staging revision are not yet
verified.

## Risks

- The enrollment method changes. It must pass the live qualification
above before broad rollout claims.
- Existing encrypted device state and the vault master key must survive
a restart or redeploy. Device expiry or removal still requires
reconnecting.
- The existing one-serving-process-per-instance constraint remains.
Concurrent replicas need durable network ownership.
- Pending setup uses bounded progress requests and an event-driven CLI
with a deadline. It adds no idle database scan.

## Model Used

OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh
reasoning. Context window size was not exposed by the runtime. No
subagents were used.

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] My branch name describes the change and contains no internal
ticket or instance details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 17:38:55 -08:00
Devin FoleyandPaperclip 493519fc88 fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (runnerd) relays provider events to the server and
waits for cumulative ACKs before it polls the provider again.
> - After a control frame, runnerd uses a 1 ms queued read only when the
executor reports a live provider poll. Otherwise it waits up to the 250
ms runtime read timeout, and any frame that arrives in that time skips
the poll.
> - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex
through ACPX and others) always wait the full 250 ms after the last ACK
of every batch, and steady control traffic starves their provider polls.
> - This pull request lets an ACPX executor report a live poll while its
restored session has an active turn.
> - The benefit is that ACPX output is relayed without the 250 ms idle
wait, and tool-heavy turns no longer starve provider polls.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents that made about 190 tool calls
in about 10 minutes. On the server, consecutive runner events arrived at
least 576 ms apart (median about 1.5 seconds), one event at a time,
while the time from each event's `emittedAt` to its insert was 0 to 0.1
seconds. The runner fell several minutes behind the provider. The
backlog then overflowed the transport's event buffer (fixed separately
in #21524) and the turn failed.

**Expected behavior**
A live ACPX turn is polled again as soon as its outstanding events are
acknowledged and no control frame is queued, the same as a Codex facade
turn.

**Steps to reproduce**
1. Run an ACPX turn that produces events while the controller sends
commands and ACKs.
2. After each ACK, runnerd calls `receive_json`, which waits up to 250
ms before it polls the provider. A frame that arrives in that time skips
the poll.

**Relevant logs or output**
```
min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched
insert time - emittedAt: 0.0 to 0.1 s
```

Related: #21524 bounds the receive buffer that this backlog overflowed.

## What Changed

- `acpx_provider_backend.rs`:
`AcpxCommandExecutor::provider_poll_is_live` returns true only when the
session was restored, has no persistence error, does not wait for a
next-turn generation, and its state is `turn_starting` or `turn_active`
with an active turn, no unconfirmed provider exit, and the run it owns.
- `native_provider_backend.rs`: `SelectedExecutor` forwards
`provider_poll_is_live` to the ACPX executor.
- `tests/acpx_warm_attachment.rs`: new test through
`NativeProviderCommandExecutor`. The session is not live before
`run.prepare` or after `session.open`, is live after `turn.start`, and a
new executor that has not restored its session is not live.

What does not change:
- runnerd still polls only when the read returns no frame, so a queued
stop or suspend still runs before the next provider poll.
- The ACK gate (no poll while sent events are not acknowledged) does not
change.
- The 1 ms read is used only while the source sequence advanced since
the last quiet poll, as for Codex today, so an idle turn does not
busy-loop.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_warm_attachment` (in
`packages/paperclip-runner`): 7 passed. The new test fails without the
new method, because the default returns false.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,127 passed, 3 failed. The failures are
`codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted") and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). They also fail without this change in the macOS sandbox
where I ran the suite, and none of them uses the ACPX executor. CI
covers them on Linux.
- The existing `durable::tests::ack_release_poll` cases cover the runner
side of the live read: an ACK releases the next poll without the read
timeout, and a command queued behind the ACK runs first.
- I did not run a live provider. After the deploy, the gap between
runner events of an ACPX run with queued output should drop below 250
ms.

## Risks

- Low risk. The change only selects the existing 1 ms queued read for
ACPX during an active turn.
- A turn that produces events continuously now polls more often. Each
poll still writes the same durable state as before, so disk writes per
event do not change.
- Analysis also found that runnerd waits for every sent event to be
acknowledged before it polls again. A bounded in-flight window would
raise throughput further, but it changes the tested rule that control
queued behind an ACK runs before the next poll. That is left out of this
pull request on purpose.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:46:05 -08:00
Devin FoleyandPaperclip 254744f77c fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.

**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.

**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.

**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```

No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.

## What Changed

- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.

## Risks

- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:58 -08:00
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00
Nicky LeachandClaude Opus 5.5 1332ab987f feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The Runner, its sidecar and the controller speak two protocols with
schemas: PRP between the controller and the Runner, and the sidecar
protocol between `paperclip-runnerd` and the ACPX sidecar.
> - A turn that the provider starts needs new fields and one new command
in those protocols.
> - This pull request declares them and generates the readers again. It
changes no behavior.
> - The benefit is that the next two pull requests show behavior only,
with the contract already in review.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 4. Its base branch is `feat/claude-provider-cycle-stream`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production. The protocol schemas.

**Subsystem affected**

Paperclip Runner: `protocol/` and the generated TypeScript and Rust
contract code.

**Current behavior**

- PRP has no way to say that a turn was started by the provider, or that
the Runner waits for such a turn.
- The sidecar protocol has no command to arm the sidecar and no event
for a turn that the provider starts.

**Proposed behavior**

- PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns`
(boolean) in the payload of `turn.start`, `origin: "provider"` on
`turn.started`, and `awaitingProviderTurn: true` on `turn.completed`.
`command-v3` and `event-v3` refer to it.
- Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the
command `session.provider_turn` with `{ turnId, armed }`, the event
`runtime.turn_started` with `{ origin: "provider" }`, and the payload of
the diagnostic for a lost provider. The schema enforces that payload for
a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`.
Diagnostics with other codes keep their open payload.
- `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No
sidecar sends it yet. The Runner of today refuses an unknown event too.
- No controller, Runner or sidecar uses the new fields in this pull
request.

**Reason and benefit**

The repository rule for Runner contracts says that new fields must be in
the schema with generated readers. A separate pull request lets a
reviewer read the contract alone.

## What Changed

- `protocol/schemas/provider-turn.schema.json` (new),
`command-v3.schema.json`, `event-v3.schema.json`,
`protocol/provider-schemas/acpx-sidecar.schema.json`,
`protocol/manifest.json`.
- `scripts/generate-acpx-sidecar-contract.mjs` and
`scripts/generate-protocol-schema-module.mjs`.
- Generated: `src/drivers/acpx/generated-sidecar-contract.ts`,
`src/protocol/generated/schema-bundle.ts`,
`src/protocol/generated/standalone-validators.ts`,
`runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`.
- `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the
Rust `match` on the event type must name the new event to compile.
- `test/protocol-contract.test.mjs`.
- The contract digests of the pack change (`prp.acpx-sidecar`,
`prp.command-v3`, `prp.event-v3`, `rust`, and the new
`prp.provider-turn`). No stored file refers to these digests. No
provider `commandDigest` and no profile version changes.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed. The contract tests: 32 passed.
- The sidecar suite and the tracker tests: 141 passed.

Not run: the full suite. CI supplies that coverage.

## Risks

- **The pack contract digest changes.** Older image packs are not used
again.
- **The opt-in field is accepted by the schema but has no effect yet.**
A controller that sends it before part 5 gets the behavior of today.
- **State saved by the previous release:** this pull request changes no
persisted shape.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:20:45 +00:00
Nicky LeachandClaude Opus 5.5 e5ad224cc5 feat(acpx): give Claude's own cycles to the adapter as one ordered stream (#21421)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - Claude Code starts a cycle of its own when a background task
finishes and no prompt is in flight. The Runner sees the raw messages of
that cycle.
> - #21293 only marks the background state as unknown when it sees such
activity.
> - This pull request gives each such cycle to the caller of the adapter
as one ordered stream: started, its events, ended with its result.
Nothing reads the stream yet.
> - The benefit is one place that knows where a cycle starts and ends,
with cycles that cannot overlap.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 3. Its base branch is
`feat/acp-session-update-event-converter`. After each part, a production
run does the same as today, because no controller sends the opt-in. Refs
#19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. How the ACPX adapter sees the cycles that
Claude starts by itself.

**Subsystem affected**

Paperclip Runner: `src/drivers/acpx` (the adapter, the runtime host, the
Claude background task tracker). Claude only.

**Current behavior**

- The raw ACP observer gives each message to the background task
tracker.
- Activity of Claude with no client request in flight makes the
background state unknown. The messages of that cycle are not kept.

**Proposed behavior**

- `ClaudeProviderCycles` puts each cycle on one ordered stream. A cycle
starts at the first update of the own session with no client request and
no prompt in flight. It ends at the `usage_update` that has
`_meta._claude/origin`.
- A caller gets the stream with `providerCycles()`, reads it with
`next()`, and calls `adopt()` or `drop()` for each cycle. A permission
request waits for the decision of its own cycle. The stream records the
cycle of each tool call, and `decisionFor(toolCallId)` gives that
decision. The binding that a caller gives to `adopt()` can also carry
`onPlanProposal`, so a plan that an adopted cycle proposes goes to the
caller as it does for a normal turn.
- With no caller, a cycle is declined and the background state becomes
unknown, as before. The sidecar does not call `providerCycles()` in this
pull request, so production does the same as today.
- The tracker reports `{ live: N }` only while N shells are running, `{
live: 0 }` only if the process never had a background task and Claude
never acted between prompts, and unknown in each other case. A dropped,
failed or cancelled cycle makes the state unknown.
- An empty result that directly follows the end of a cycle (zero usage,
same context use and cost) is ignored. Claude Code sometimes sends it.
It must not become a cycle, and its usage must not be reported two
times.

**Reason and benefit**

- One producer and one reader make the order of cycles a property of the
structure.
- The tracker does not read the order of messages to decide which
finished task a cycle reported on. Captures showed that Claude Code can
put two notifications in one cycle, start a second cycle, or give a
notification to a cycle in progress, and the ACP stream does not say
which.

## What Changed

- `src/drivers/acpx/claude-provider-turns.ts` (new, 303 lines):
`ClaudeProviderCycles`.
- `acpx-runtime-adapter.ts` (78 added, 8 removed) and `runtime-host.ts`
(24 added): `providerCycles()` and `providerTurnsAvailable()`, and the
permission path that waits for the decision.
- `claude-background-tasks.ts` (16 added, 7 removed): the rule above.
- Tests (764 lines): `claude-provider-turns.test.ts`,
`claude-provider-turn-tasks.test.ts`, `claude-background-tasks.test.ts`,
a permission test in `acpx-runtime-adapter.test.ts`, two tests with the
real `claude-agent-acp`
(`acpx-runtime-adapter.claude-provider-turn.test.ts`,
`acpx-runtime-adapter.claude-overlapping-shells.test.ts`), and a
captured raw stream as a fixture.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- 10 test files, 294 passed. This includes the test of #21293 with the
real `claude-agent-acp`, not changed: with no caller of the stream it
still gets `{ live: 1 }` at the end of the turn and an unknown state
after the cycle of Claude.
- Real `claude-agent-acp` with a local fake Anthropic API: a background
shell finishes, and the stream gives one cycle with its own usage. With
two shells where the second finishes before the first cycle sends its
first update, the first cycle ends with an unknown state.
- Mutation checks, each made the named tests fail: an end item before
the last message of a cycle; a drop that does not make the state
unknown; a finished task that reports `{ live: 0 }`; the empty result
not ignored; a permission request that does not wait.
- Live run with a real Claude account, not in CI, on the full change:
one background shell gave `{ live: 1 }`, one cycle, then unknown. Two
shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown.

Not run: the full suite. CI supplies that coverage.

## Risks

- **The time at which the state becomes unknown changes a little.** With
no caller, a cycle makes the state unknown at its first message. Before,
it was at the first message of an activity kind. The result for the
Runner is the same: a restart at the end of the turn.
- **Limit, by design.** After a cycle, `{ live: N }` counts only running
shells. A command that finishes while Claude already reports on an
earlier one gets no cycle that the Runner waits for.
- **The rule for the empty result is narrow** and comes from captures of
one Claude Code version. The reason that Claude Code sends it is not
known.
- **A queue of 4,096 messages for one cycle.** A cycle that is larger
fails, and the state becomes unknown.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:48:53 -08:00
Nicky LeachandClaude Opus 5.5 1140984b82 feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.

**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.

**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.

**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.

## What Changed

- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.

## Verification

- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.

## Risks

- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:43:10 -08:00
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
Nicky LeachandClaude Opus 5.5 1d112a504f feat(acpx): convert raw Claude session updates the way ACPX does (#21420)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The Runner sees each raw ACP message before ACPX does. A cycle that
Claude starts by itself is not a part of an ACPX prompt, so ACPX gives
no events for it.
> - To show that cycle in the same form as a normal turn, the Runner
must convert the raw messages itself. ACPX 0.22 does not export its
converter.
> - This pull request adds a copy of that converter with a test that
compares it with ACPX. Nothing uses it yet.
> - The benefit is that a later pull request can show the cycle of
Claude with the same events as a normal turn.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 2. Its base branch is
`refactor/acpx-sidecar-runtime-event-step`. After each part, a
production run does the same as today, because no controller sends the
opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing yet. It adds a module that a later part uses.

**Subsystem affected**

Paperclip Runner: `src/drivers/acpx`. One new direct dependency in the
Runner and the server.

**Current behavior**

- ACPX converts `session/update` messages to runtime events inside a
prompt. The function is private.
- The Runner has no way to make the same events from a raw message.

**Proposed behavior**

- `acp-session-update-event.ts` converts a raw `session/update` to the
event that ACPX 0.22 makes for the same update.
- It validates the message with `@agentclientprotocol/sdk` 1.9.0. This
is the version that ACPX 0.22 already installs.
- No production code imports the module in this pull request.

**Reason and benefit**

A copy with a comparison test is a smaller risk than a patch of ACPX.
The test fails when the two differ.

## What Changed

- `src/drivers/acpx/acp-session-update-event.ts` (new, 367 lines).
- `acp-session-update-event.test.ts` and
`acpx-runtime-adapter.claude-session-update-events.test.ts` (new, 363
lines).
- `packages/paperclip-runner/package.json` and `server/package.json`:
`@agentclientprotocol/sdk` 1.9.0 as a direct dependency. The server
needs the line because it copies the Runner build with no
`node_modules`.
- `pnpm-lock.yaml` is not in this pull request. The lockfile workflow
owns it.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- The two new test files and the sidecar suite: 130 passed.
- The comparison test runs the real ACPX runtime with the pinned
`claude-agent-acp` and a local fake Anthropic API, and gives the same
raw updates to the copy. The events are the same.
- A second test fails if the SDK version of the Runner is not the
version that ACPX resolves. With 1.4.0 in `package.json` it fails.

Not run: the full suite. CI supplies that coverage.

## Risks

- **A new direct dependency.** It is already installed through ACPX, so
no new package is downloaded. The version test keeps the two the same.
- **The converter is a copy of ACPX code.** An ACPX upgrade must run the
comparison test.
- **The module is not used yet.** It is used in part 3.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:14:59 -08:00
Nicky LeachandClaude Opus 5.5 055a0a3c54 refactor(acpx-sidecar): extract the runtime event step and the test process helper (#21419)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The last part of that stack adds a second kind of turn to the
sidecar. It needs two pieces of the code of a normal turn.
> - This pull request moves those two pieces into functions, and moves a
test helper to a file of its own. It changes no behavior.
> - The benefit is that the later pull request shows only new behavior,
not moved code.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 1. Its base branch is
`refactor/remove-direct-gateway-path`. After each part, a production run
does the same as today, because no controller sends the opt-in. Refs
#19912 (design).

**What existing behavior does this improve?**

Nothing for a user. The structure of the ACPX sidecar code.

**Subsystem affected**

Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts` and its test.

**Current behavior**

- `pumpTurn` has the code that prepares each runtime event for the
Runner, and the code that sends the terminal frame of a turn, inline.
- The sidecar test file has the `SidecarProcess` helper that starts a
sidecar process.

**Proposed behavior**

- `createRuntimeEventStep` has the event code. `emitTurnTerminal` has
the terminal code. `pumpTurn` calls both.
- `src/test-support/acpx-sidecar-process.ts` has the helper, so a second
test file can use it.
- No protocol, event or timing change.

**Reason and benefit**

A later pull request adds a turn that Claude starts by itself. It uses
the same two functions, and its tests use the same helper.

## What Changed

- `src/cli/acpx-runtime-sidecar.ts`: 74 lines added and 56 removed. The
code moved with its comments, with no change.
- `src/cli/acpx-runtime-sidecar.test.ts` and
`src/test-support/acpx-sidecar-process.ts`: the helper moved, 138 lines
added and 132 removed.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- `npx vitest run src/cli/acpx-runtime-sidecar.test.ts`: 125 passed. The
tests are the same as before, apart from the import of the helper.

Not run: the full suite. CI supplies that coverage.

## Risks

- **A refactor of the turn loop of each ACPX agent.** The moved code
runs for Claude, Codex, Pi, Kimi, Cursor, Copilot and Grok. The existing
sidecar suite covers it. No new test is necessary, because no behavior
changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:39:55 -08:00
DottaandPaperclip 0ead74b4f3 feat(slack): show useful progress without extra messages (#20827)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Chat connectors show agent work in the source conversation.
> - Slack progress uses a small set of general status messages.
> - Native runtimes already mark some completed assistant notes as
public progress.
> - The server can select those notes and control their size and rate.
> - This pull request keeps one progress message and sends the final
answer as a new reply.
> - Users can follow useful changes without reading each tool event.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Automatic progress updates in Slack task threads.

**Current behavior**

Users see general messages such as “Maya is making progress”. The first
answer can replace that message. A source update that arrives during the
progress delay can be lost if no later event arrives.

**Proposed behavior**

Show the latest short public note in one message. Wait 10 seconds before
the first update and at least 30 seconds between updates. Recover
pending work after a restart. Send the final answer as a fresh reply.

**Reason and benefit**

Users can see what the agent checks and what it found. The thread stays
short. The model needs no new tool call or summary model.

Related change: [resolved-card chat
bindings](https://github.com/paperclipai/paperclip/pull/20825). This
branch includes that master revision. The progress change uses the
existing source authority checks.

## What Changed

- Select completed `agentMessage` items from the normalized PRP
`progress` channel. Keep unknown channels, reasoning, tool output, and
raw provider fields out of progress text.
- Sanitize each note. Keep at most two sentences and 400 characters. Add
a task link when a public board URL is available.
- Prefer public notes over general tool status. Remove repeated text and
combine bursts into the latest pending note.
- Use run events and the publication outbox as saved state. Use the
shared deadline worker for the grace period and update delay. A delivery
receipt wakes the worker when an earlier update is still in flight.
Settled history does not poll.
- Apply the run's registered-secret filter before text selection or
storage. Reuse the latest usable note within a sweep. Invalidate the
cache when the source event or secret registry changes.
- Reuse the exact source-bound Slack fast-response receipt when it
replaced the working placeholder. Check its current outbound message
link before editing it.
- Send Slack answers as new replies. After delivery, queue a separate
cleanup edit for a generic working message. Keep its current outbound
link as the edit authority. Leave useful public notes in place. Cancel
pending public progress when an answer, question, or terminal run takes
over. Keep the existing delivery policy for other connectors.
- Add one short Slack prompt instruction. Update product documentation
and regression tests.

The Slack module owns public-note selection, cadence, lane eligibility,
and cleanup. The shared progress service holds its existing issue and
run locks and calls a narrow provider policy. The secret registry
supplies redaction and a version token for the per-sweep note cache. The
existing outbox owns delivery, retries, and authority checks. The Slack
module owns receipt reuse. Shared delivery changes supply the receipt
wake and select the Slack reply policy. No schema or configuration
change is required.

## Verification

- The initial content-selection regression failed before the
implementation and passed after it.
- The follow-up reproduced two review defects before the fixes. A
registered synthetic secret appeared in saved progress. A generic
working message stayed active after the answer. Both regressions now
pass.
- The final content, database-worker, query, protocol-boundary, and
redaction selection passes 78 tests across four files.
- The earlier affected transport selection passes all 65 cases. It
covers public progress, native cadence, separate answers, Slack and
Telegram retry isolation, FIFO restart, and cleanup ownership. Its first
run passed 64 cases and found one old assertion that rejected the
intended cleanup edit. The corrected assertion checks the exact
successor message ID.
- Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm
--filter @paperclipai/server typecheck`. An earlier attempt found two
transaction type errors. The helper contracts now accept their actual
database operations. The final run passes all source and test layers.
- The broad Slack selection stopped after 189 passing cases when the
sandbox killed its worker with SIGKILL. That attempt is not a pass. The
focused transport run above supplies affected local coverage. Required
CI must supply broad coverage on the final head.
- `pnpm verify --explain` returns exit 2 because the product
documentation change requires broad CI coverage. No full local suite was
started.
- Preflight installed dependencies and built the SDK. Cargo is absent,
so local Rust checks could not run. No Rust source changed.
- [CI on
`02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242)
passed build, typecheck, policy, all browser shards, Runner/Rust checks,
and the other test lanes. One chat shard failed in three Slack cases.
Each expected one edit after an answer was already sent. The intended
cleanup adds a “Response sent.” edit before the later failure notice.
All three failures reproduced locally. The test assertions now check
both exact edit texts and the original run's message ID. They also check
that another sweep adds no edits.
- The latest local run passes all 85 selected transport cases. It
includes all 18 Slack/Telegram answer and failure orders, plus progress,
deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm
exec vitest run server/src/__tests__/chat-channels.integration.test.ts
-t 'preserves selected answer and failure lanes|public progress|native
progress|keeps intermediate.*agent comments|keeps
a.*retry.*queue|FIFO'`.
- [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545)
passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed
jobs passed. This includes build, typecheck, general tests, serialized
tests, all eight browser shards, Runner/Rust checks, and the required
aggregates. The affected chat shard now passes. The test-file selection
job was skipped because this run used full coverage.
- [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124)
completed on the same head with 5/5 and no new findings. All four inline
review threads are resolved. The review job and Socket checks passed. Of
61 check runs, 57 succeeded, three were skipped, and one was neutral. No
checks remain pending or failed. The PR is mergeable with a clean state.
- [Superagent Security
Scan](https://github.com/paperclipai/paperclip/runs/132142345827)
returned a neutral result with “PR scan workflow failed”. It supplied no
security verdict. Do not count it as a successful scan.
- Live Slack notification and unread behavior has not been tested.
Service tests use real PostgreSQL and a fake provider runtime.

## Risks

- Only runtimes that map commentary to the explicit public `progress`
channel supply contextual prose. Other runtimes retain general status
messages.
- Slack final answers now create a fresh reply. This is an intended
delivery change.
- Public commentary must suit the current conversation audience. Text
sanitization adds protection; it does not authorize sharing unrelated
private material.
- Source events and outbox rows supply recovery state. No new timer
table or recurring idle scan is added.
- The existing large integration suite retains its provider fixtures and
lifecycle assertions. No source file crosses the 1,000-line threshold in
this change.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact
deployment ID was not supplied.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:48:05 -06:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
DottaandClaude Opus 5.5 342fb308bd perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Text deltas (`item.delta`) are about 85% of that process's SQL
statements: each delta is one transaction of four statements on the
shared 10-connection pool
> - At 50 agents and 20 deltas per second, the process falls behind and
unrelated agent requests wait for the pool
> - This pull request commits text deltas in batches: one transaction
for all deltas that arrive within 250 ms, with one multi-row insert
> - The benefit is about 7 times fewer transactions and 5–6 times fewer
statements per delta, with no delay for the first text of an item

## Linked Issues or Issue Description

Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a
of the agent scaling plan in #21418.

**What existing behavior does this improve?**
The persistence of native text deltas. `native-session-runtime` awaited
`PaperclipControlPlanePort.appendEvent` for each event, and the port ran
one transaction per event.

**Subsystem affected**
Server run event ingest (`paperclip-control-plane-port.ts`,
`heartbeat-run-events.ts`) and the runner package's turn consumer
(`native-session-runtime.ts`).

**Current behavior**
Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In
the load model with 50 agents (deltas only), that is about 4.0
statements and 1.0 transaction per delta.

**Proposed behavior**
The turn consumer starts the next append without waiting for a delta's
receipt. The port collects deltas and commits them in one transaction
with one insert. In the same load model, that is about 0.7 statements
and 0.14 transactions per delta.

**Reason and benefit**
Fewer transactions free the shared pool and the main thread for other
agents' requests. This is the largest per-agent cost in the scaling
plan.

**Breaking changes**
None. The row shape does not change. A port without the new capability
keeps the earlier serial behavior.

## What Changed

- `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends
native source events of one run and one source in one transaction. When
the events continue the stored source cursor (from #21435), it sends
four statements for the whole batch: the run lock, one replay probe, one
`insert ... select from jsonb_array_elements(...)` of every row, and the
commit. Any other batch (a replay, a gap, an event with a wake, or a run
without a cursor) appends its events one by one in the same transaction,
with the single-append checks.
- **Storage choice: one row per delta, written by one statement per
batch.** I did not store one row per batch. The transcript, live events,
run log export, chat progress, replay (`replayEvents`), recovery cursors
and the UI all read one row per delta. One statement per batch keeps
every reader unchanged, and rows written before this change read the
same.
- `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An
`item.delta` appended with the new `streaming` option goes into a queue:
- It commits at once until a delta with text commits after the latest
other event. So the first text of each item, and the time to first
token, never wait for the timer.
- Later deltas commit after at most 250 ms, or at once when the batch
has 100 deltas.
- Every other event, `checkpointSession`, `completeRun`, `replayEvents`
and `provesNoRunProviderWork` first commit the queue. A non-streaming
event is never committed before the deltas that preceded it.
- A delta appended without `streaming` commits at once, so callers that
wait for each receipt are not slowed.
- The receipt of a delta settles only after its batch commits and its
observers (live event, executor callback) run, in order.
- If a batch fails, every delta in it rejects and nothing from it is
stored.
- `native-session-runtime` `consumeTurn` appends deltas with `streaming`
and does not wait for their receipts (at most 256 pending). Before any
other event, any result, and the governed-wait settlement, it calls
`flushStreamingEvents()` and then waits for all pending receipts, so a
boundary never waits for the timer. A failed delta stops the turn before
a later event is appended. The consumer joins every pending delta append
before it settles, also when the provider stream fails.
- If one observer of a committed batch fails, only that delta's receipt
rejects. The other rows are still published and observed.
- `ControlPlanePort` contract: optional `pipelinesStreamingEvents`,
`flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`.

## Verification

- Red/green. The new tests failed before the change and pass after it:
- Runner: "starts text delta appends without waiting and flushes them
before any other event" (`native-session-runtime.test.ts`). The port in
the test has no timer, so the turn finishes only if the boundary
flushes. Before: the appends were serial, and later the boundary waited
for the timer.
- Runner: "joins pending delta appends before a failed turn settles".
Before: the turn settled while a delta commit still ran.
- Port: "observes every committed delta of a batch when one observer
fails".
- Port: "commits the first text delta at once and later deltas in one
batch before the next other event". It checks one transaction and one
insert for five deltas, flush before a non-streaming event, receipts
only after commit, and immediate commit of the first delta with text.
Before: each delta committed alone.
- Port: "flushes queued deltas on the timer and before a replay read,
and rejects every delta of a failed batch". It checks the
crash-before-commit case: a revoked writer fails the batch, nothing is
stored, and the resent deltas commit idempotently (the stored one is a
duplicate).
- Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`:
one lock, one probe and one insert per batch; replay idempotency and
payload conflict; a batch that fails before commit stores nothing and
its resend commits; the one-by-one path for no cursor and for gaps.
- Focused suites pass: `heartbeat-run-event-atomic-append`,
`heartbeat-run-event-sequencing`, `paperclip-control-plane-port`
(including the package conformance suite),
`paperclip-semantic-conformance`, `native-observer-replay`,
`native-event-boundary-golden`, `live-recovery-work`,
`native-interaction-bridge`, `chat-native-progress-work`, and
`native-session-runtime.test.ts` in the runner package.
- `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`
and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls
the store directly, so for this run I added a local `BATCH_MS` mode that
applies the port's batching rules (first delta at once, 250 ms timer,
100 cap, flush before other events) and does not wait for delta
receipts. That mode is not in this pull request. The machine was very
busy (load average 40–60), so absolute figures are noisy; per-delta
ratios are the stable result:

| Deltas only | Statements/s | Transactions/s | Deltas done of offered |
Statements per delta | Transactions per delta | Lateness p50 / p99 |
  | --- | --- | --- | --- | --- | --- | --- |
| One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 |
1.00 | 3,074 / 5,384 ms |
| Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 /
2,858 ms |
| One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 |
3.99 | 1.00 | 6,558 / 11,464 ms |
| Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17
| 187 / 556 ms |

With all sources on, the machine was saturated (event-loop utilization
0.96–0.98) in both modes and the results were not usable (764 and 12,800
of 20,000 deltas done).

## Risks

- **Ack and crash (read first).** In the live runnerd path the runner's
ack does not wait for the `heartbeat_run_events` row today.
`DurablePrpControlPlane.#event` acks after the in-memory journal commit.
Streaming events are not in the controller checkpoint. So a delta that
is acked but not yet appended is lost if the server crashes, and the
runner does not resend it. This is the earlier behavior. This pull
request makes that window up to 250 ms longer. The final text still
survives through `item.completed`, which commits immediately and in
order. This pull request does not change `durable-prp-control-plane.ts`.
- Live text in the UI arrives in steps of up to 250 ms after the first
text of an item.
- A delta failure now surfaces at the next non-delta event or at the end
of the turn, not at the delta. The turn stops before it appends a later
event.
- The governed-wait drain and the control-event replay still append one
event at a time. They do not carry deltas.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:34:01 -06:00
Nicky LeachandClaude Fable 5.2 0aaab644fe refactor(runner): remove the direct assigned-gateway path (#21311)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent reaches the tools of its app connections through an MCP
gateway.
> - Before #21274, some native harnesses got the gateway URL and a
bearer token and connected to the gateway directly. The Runner has code
for that: an HTTP MCP server for ACPX providers, a server entry in the
Codex `config.toml`, a server entry in the OpenCode config, and the
lists that pass the three `PAPERCLIP_NATIVE_MCP_*` variables to a
sidecar.
> - #21274 gives each native harness its gateway tools through the PRP
relay. Core removes the three variables from each Runner environment.
The direct code path gets no binding.
> - Code that cannot run still needs maintenance, and it can still put a
token in a provider process if a variable arrives by another way.
> - This pull request deletes the direct path from the Runner.
> - The benefit is one path for assigned gateway tools, and no Runner
code that can give a gateway token to a provider.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is in a stack. Its base branch is
`fix/keep-quiet-claude-provider-between-turns` (#21293). It depends on
#21274, which is the bottom of the stack. On `master` without #21274,
the direct path is still in use.

**What existing behavior does this improve?**

The Runner code that connects a provider directly to the assigned MCP
gateway.

**Subsystem affected**

Paperclip Runner: the TypeScript drivers and the Rust `runner-core`
environment lists.

**Current behavior**

After #21274, Core never gives `PAPERCLIP_NATIVE_MCP_NAME`,
`PAPERCLIP_NATIVE_MCP_URL` or `PAPERCLIP_NATIVE_MCP_TOKEN` to a Runner.
The Runner still has the code that reads them and configures a direct
gateway server for ACPX providers, Codex and OpenCode.

**Proposed behavior**

The Runner has no code that reads the three variables to configure a
provider. If the variables are in the environment of a Runner, no
provider environment, sidecar, Codex `config.toml` or OpenCode config
gets them.

**Reason and benefit**

Less code, one path, and no way for a gateway token to reach a provider
process through the Runner.

## What Changed

- Deleted `packages/paperclip-runner/src/drivers/native-mcp.ts`
(`nativeMcpLaunchBinding`).
- `drivers/acpx/runtime-host.ts`: removed the native gateway server
entry and the name conflict check. The admission gate keeps its message.
A run with a binding id is admitted when the sidecar uses the PRP
transport with its tool bridge, as the production sidecar always does.
- `drivers/acpx/environment.ts`: the name and the URL are not in the
provider allowlist.
- `drivers/runtime-context-materializer.ts` and
`live/runnerd-codex-transport.ts`: the Codex home functions have no
`nativeMcp` parameter. The sidecar token pass-through is gone.
- `drivers/opencode/opencode-server-driver.ts`: removed the read of the
binding, its trace redaction value, and the gateway server entry in the
OpenCode config.
- `drivers/stdio-mcp.ts` and `runner-core/src/stdio_mcp.rs`: removed the
reservation of the gateway server name. No gateway server exists, so no
name can conflict. The names `paperclip` and `paperclip_runner_*` stay
reserved.
- `runner-core/src/acpx_sidecar_transport.rs` and `codex_provider.rs`:
the three variables are not in the pass-through lists.
- `control-plane/durable-prp-control-plane.ts`: the three variables are
not in the platform environment list.
- `packages/paperclip-runner/README.md`: the text about direct callers
and assigned servers.
- Kept: the three names stay in the reserved lists
(`src/configured-environment.ts`,
`runner-core/src/configured_environment.rs`). A user cannot configure
them. Tests in both languages now check all three names.
- No server code changed. The relay needs the gateway token and URL on
the control plane.

23 files, 73 lines added and 211 lines removed.

## State saved by the previous release

- This pull request changes no persisted shape. The provider server
lists, the launch profile digests, the protocol versions and the sidecar
contract are the same as with #21274.
- A Runner or sidecar that the previous release started keeps its own
code until its run ends, as #21274 describes.
- A kept Codex home that has the old gateway server in `config.toml` is
written again at the next start and at a recovery of the same run. A
test covers this.
- If the old variables reach a new Runner, no provider gets them. Tests
cover this for ACPX, Codex and OpenCode.

## Verification

From `packages/paperclip-runner`:

- `npx vitest run src/drivers/acpx/runtime-host.test.ts
src/live/runnerd-codex-transport.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/opencode/opencode-server-driver.test.ts
src/live/runnerd-codex-transport-warm-attachment.test.ts
src/drivers/runtime-context-materializer.test.ts
src/drivers/agent-observability.test.ts src/drivers/stdio-mcp.test.ts
src/control-plane/durable-prp-control-plane.test.ts
src/configured-environment.test.ts`: `10 passed (10)` files, `807
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: no
errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes.
- `cargo test -p paperclip-runner-core --test acpx_sidecar_transport`:
20 passed. `cargo test -p paperclip-runner-core --lib`: 639 passed, 4
ignored.

Tests that failed with the old code and pass now:

- `runtime-host.test.ts`: "admits a relayed … run with a binding id and
ignores a previous-release gateway binding", for Claude, Codex and
Cursor.
- `runnerd-codex-transport.test.ts`: "drops a previous-release gateway
binding from the … runner environment", for ACPX Claude, Codex, Pi,
Cursor, Copilot and Kimi, and for OpenCode.
- `opencode-server-driver.test.ts`: the OpenCode config has only the
task bridge.
- `runnerd-codex-transport-warm-attachment.test.ts`: a restored Codex
home has no gateway server and no gateway credential in `config.toml`.
- `environment.test.ts`: a provider environment has none of the three
variables.
- Rust
`previous_release_gateway_binding_and_unrelated_secrets_never_reach_the_sidecar`:
a Runner that starts with the old variables gives none of them to the
sidecar, for five agents.

Mutation check: with the three names put back in the Rust pass-through
list, the Rust test fails.

Not run: a live run with a real provider. Not run: `pnpm verify` and the
full suite. CI supplies that coverage.

## Risks

- **This pull request is safe only after #21274.** The stack order keeps
that.
- **A developer who sets the three variables by hand for
`eval-session`** no longer gets direct gateway tools. No document
describes that use.
- **The warm attachment test uses the staged Runner binary from
preflight.** That binary does not have the Rust changes of this pull
request. The Codex path of that test does not use the changed lists.
- The evidence that no entry point supplies a binding comes from a read
of the code: Core removes the variables before it builds each Runner
environment, and a native run cannot use the in-process path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, and ran the verification and a mutation check
again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), traced each code path and
wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 13:33:47 -08:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
DottaandClaude Opus 5.5 2a8b485140 fix(runner): accept object fields sent as JSON text and name wrong-type rejections (#21485)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner checks each tool call from the model against
the tool's input schema before it sends the call to the control plane
> - Some model hosts send an object argument as its JSON text. On the
DigitalOcean host, `paperclip_finish` arrived with `continuation` as a
string
> - The runner rejected these calls, and the repeated-rejection guard
then told the model that its arguments "did not arrive", which is wrong
when every field arrived with the wrong type
> - This pull request makes the runner parse such a string back into its
object when that is safe, records each coercion as a run event, and
makes the guard name the actual problem
> - The benefit is fewer stuck finish calls, and guidance that points
the model at the real fault

## Linked Issues or Issue Description

Refs #21398

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
On the DigitalOcean host, 6 of the 8 remaining "Invalid tool input"
rejections were `paperclip_finish` calls with `continuation` sent as a
JSON string (`/continuation must be object,null`). Examples:
`gateways-profiles.rotate-token`, `instance-operations.restart-service`,
and `questions-and-approvals.multi-part-question`. The run diagnostic
recorded `continuationShape: "string"`. After the second identical
rejection, the guard from #21398 said "If you meant to send other
arguments, they did not arrive". Every argument had arrived; one had the
wrong type.

**Expected behavior**
When a schema field is object-typed, does not allow text, and the text
parses to a plain JSON object that makes the call valid, the runner
accepts the call with the parsed object and logs a run event. When a
call stays invalid, the guard says which required fields are missing and
which fields have the wrong type.

**Steps to reproduce**
1. Start the OpenCode MCP bridge.
2. Call `paperclip_finish` with `reportedWorkDisposition: "yielded"` and
`continuation` as `JSON.stringify({ kind: "response_wake", ... })`.
3. Before this change, the bridge returns `Invalid tool input:
/continuation must be object,null`.

## What Changed

- Added `packages/paperclip-runner/src/drivers/tool-input-coercion.ts`.
`admitToolInput` validates the arguments. If they are invalid, it parses
a string in an object-typed field back into its object. It does this
only when the field's schema declares `object`, does not allow `string`,
the text parses to a plain JSON object, and the whole coerced call
validates. It applies to any object-typed field at any depth, not only
`continuation`.
- The OpenCode MCP bridge and the runner tool bridge use
`admitToolInput`. They dispatch the coerced arguments, and they send one
`paperclip.runner.tool-input-coerced/v1` diagnostic per accepted
coercion. The OpenCode driver already records bridge diagnostics as run
events. The diagnostic names the tool and the field paths. It never
contains the values. The runner tool bridge has a new optional
`onDiagnostic` option.
- When parsed text is still invalid, the rejection describes what is
wrong inside the parsed object, for example `/options/mode must be
string`.
- The repeated-rejection guard now says "Required fields X did not
arrive" only for missing required fields. For fields with the wrong
type, it says "Every field you sent arrived, but one has the wrong type:
continuation must be an object or null, not text".

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts` (63 tests pass).
- Red/green: with the source changes reverted and the new tests kept, 5
tests fail: coercion acceptance and its run event (both bridges), no
coercion for text fields or non-object text, the wrong-type guard
wording, and the missing-field guard wording. With the change, all of
them pass.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.
- A narrow paid run of `questions-and-approvals.multi-part-question`,
`runs-adapters.inspect-failed-run` and `budgets-costs.explain-hard-stop`
ran on a branch that combines this PR with the two sibling cohort-13
PRs. The results are in a PR comment.

## Risks

- Low risk. Coercion runs only after a call fails validation, and the
call is accepted only if the coerced form validates. A field that allows
text is never parsed, so a valid string never changes meaning.
- Previous-release story (upgrade-safe run state): this change adds no
persisted run, session, launch or pack state. It changes only how one
in-flight tool call is admitted. Saved state carries forward unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:06:37 -06:00
Nicky LeachandClaude Fable 5.2 0ebe6d846d perf(runner): keep a quiet Claude provider between turns of a run (#21293)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar. The
Runner stops and starts the sidecar and Claude at each turn boundary.
> - The reason was that a late tool call from an earlier turn could run
in a later turn. #21278 removes that reason for Claude: the sidecar
refuses a call that the active turn did not announce.
> - A second reason remains. Claude Code works by itself between
prompts. When a background task finishes, Claude starts a cycle of its
own. If the process stayed alive with a live background task, that cycle
could start between turns or mix with the next turn.
> - So the Runner can keep the process only when it knows that Claude
has no background work.
> - This pull request adds that knowledge and uses it. After a Claude
turn that completed with no background work, the Runner keeps the
process for the next turn of the same run.
> - The benefit is that a later turn of the same run starts with no
provider restart. This is also the base for the next step, which lets a
run continue when background work finishes.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is stacked on #21278. Its base branch is
`fix/bind-claude-tool-calls-to-turn`. Refs #19912 (design), #12419
(restart at turn start), #20145 (restart at park) and #20190 (restart at
turn end).

**What existing behavior does this improve?**

The provider restart between turns of the same run, for Claude through
ACPX.

**Subsystem affected**

Paperclip Runner: the ACPX sidecar (TypeScript) and the ACPX provider
session in `runner-core` (Rust). Claude only.

**Current behavior**

- When a completed turn settles, the Runner restarts the sidecar and
Claude (`restart_session_after_turn`, #20190).
- If that restart did not run, the next turn restarts them at its start
(`start_turn`).
- The Runner does not know if Claude has background tasks.

**Proposed behavior**

- The sidecar asks `claude-agent-acp` for its async task reports. This
is an optional capability of the pinned `claude-agent-acp` 0.73.0. The
patch does not change.
- A tracker reads those reports and the id of each background Bash
command. It reports `{ live: 0 }` only when it knows that this process
never had a background task and that Claude did not act between prompts.
In all other cases the state is unknown.
- The sidecar adds `backgroundTasks` to the terminal event of a
completed Claude turn, and to `session.snapshot`.
- The Runner keeps the process between two turns only when all of these
are true:
  - the agent is Claude;
  - the turn completed (not failed, cancelled or interrupted);
  - the terminal event reported `backgroundTasks: { "live": 0 }`;
  - each existing condition of the turn-end restart is true;
- before the next turn, a new `session.snapshot` reports the same
identity and run, no active turn, no pending work, and `{ "live": 0 }`;
  - the settled-turn ledger and the tool replay ledgers are not full.
- In each other case the Runner restarts as before.
- A kept process keeps its tool call-id tombstones and its settled turn
identities.

**Reason and benefit**

- A turn that follows a quiet turn in the same run starts without a
restart.
- The decision is safe by default: unknown state means a restart.

## What Changed

TypeScript (`packages/paperclip-runner/src`):

- `drivers/acpx/claude-background-tasks.ts` (new):
`ClaudeBackgroundTaskTracker`. It reads raw ACP messages. It reports
unknown when the capability is not negotiated, when a task finished,
when a live task is not a shell command, when an update is malformed,
when a limit is reached, when Claude acts with no client request in
flight, or when Claude calls `CronCreate` or `ScheduleWakeup`.
- `drivers/acpx/provider-failure.ts` and `acpx-runtime-adapter.ts`:
Claude sessions also advertise the `asyncTasks` capability. The raw ACP
observer gives each message to the tracker. ACPX 0.22 drops the
`async_task_*` updates before the turn event stream, so only the raw
observer sees them.
- `drivers/acpx/runtime-host.ts`: `backgroundTaskState()`.
- `cli/acpx-runtime-sidecar.ts`: the terminal event of a completed
Claude turn carries `backgroundTasks` in the same payload as
`goalAtTerminal`, with the same size limit. `session.snapshot` carries
it when no turn is active.
- `drivers/acpx/claude-tool-use-admission.ts`:
`hasRoomForAnotherTurn()`. When the admitted-id ledger cannot hold one
more full turn, the sidecar reports unknown, so the Runner restarts
before the ledger is full.

Rust (`packages/paperclip-runner/runner/crates/runner-core`):

- `acpx_provider_session.rs`: the session records a quiet turn from the
terminal event. `start_turn` continues in the same process only under
the conditions above, and then uses `prepare_turn`, which keeps the
tombstones.
- `acpx_provider_generation.rs`: `can_continue_after_turn`,
`continue_after_quiet_turn`, and the snapshot check
`background_work_is_quiet`.
- `acpx_park_preparation.rs`: the turn-end restart does not run for a
quiet Claude turn.
- `acpx_provider_backend.rs`: `session.reconciled` reports a replaced
process only when the turn start replaced it.
- `bin/fake-acpx-sidecar.rs`: new option `--background-tasks` for tests.
- `runner/DURABLE_TRANSPORT.md`: the rule for a kept sidecar.

Contract (`packages/paperclip-runner/protocol`):

- The `backgroundTasks` report is `$defs.backgroundTasks` in
`provider-schemas/acpx-sidecar.schema.json`: an object with only `live`,
a non-negative integer. The schema validates it on
`runtime.turn_terminal`.
- `scripts/generate-acpx-sidecar-contract.mjs` generates the TypeScript
type and reader and the Rust reader from it. The tracker, the sidecar
and the Rust session use the generated code.
- `fixtures/provider-boundary/acpx-background-tasks.json` has shared
test documents. The TypeScript reader, the Rust reader and the schema
are tested with the same accepted and refused shapes.
- This changes the contract digest of the provider pack
(`prp.acpx-sidecar` and `rust`). Older image packs are not reused. No
provider `commandDigest` and no profile version changes.

Not changed: park and run attachment. A process that ran a turn is still
restarted at park, and only a process that ran no turn is bound to the
next run. Other ACPX agents and Codex do not change. The
`claude-agent-acp` patch does not change.

## State saved by the previous release

- This pull request adds no persisted state. The three new fields of the
session are in memory only.
- A terminal event or a snapshot without `backgroundTasks` means
unknown. An older sidecar therefore gets the restart, as before.
- An older Runner with a new pack ignores the field and restarts at each
turn, as before.
- After a Runner restart, the in-memory state is gone. The next turn
restarts the provider, as before. A test covers this.

## Verification

From `packages/paperclip-runner`:

- `npx vitest run src/drivers/acpx/claude-background-tasks.test.ts
src/drivers/acpx/acpx-runtime-adapter.claude-background-tasks.test.ts
src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts
src/drivers/acpx/claude-tool-use-admission.test.ts
src/cli/acpx-runtime-sidecar.test.ts
src/drivers/acpx/acpx-runtime-adapter.test.ts
src/drivers/acpx/runtime-host.test.ts`: `7 passed (7)` files, `363
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p
tsconfig.test.json`: no errors.
- `pnpm run typecheck:typescript` (it includes the contract checks): no
errors. `node --test test/acpx-sidecar-contract.test.mjs`: 17 passed.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes.
- `cargo test -p paperclip-runner-core --test
acpx_quiet_turn_continuation --test acpx_park_preparation --test
acpx_park_readiness --test acpx_provider_turns --test
acpx_provider_session --test acpx_warm_attachment --test
native_provider_backend`: 4, 33, 8, 31, 12, 6 and 47 passed. No test
failed.
- `cargo test -p paperclip-runner-core --lib`: 637 passed, 4 ignored.
- Mutation check: when the Rust code accepts each live count as quiet,
two tests in `acpx_park_preparation` fail.

New tests:

- **Real ACP runtime**
(`acpx-runtime-adapter.claude-background-tasks.test.ts`), with the
pinned `claude-agent-acp`, the pinned Claude Code client and a local
fake Anthropic API. A foreground Bash turn reports `{ live: 0 }`. A
background Bash in the same process reports `{ live: 1 }` at the
terminal event, and unknown after Claude ran its own cycle for the
finished task. Each turn has its own usage receipt. One process serves
all turns.
- **Tracker unit tests** (13): spawn, completion, failure, stop, pause,
a task that is not a shell, a subagent, no negotiation, malformed
updates, activity between prompts, scheduling tools, session load
replay, limits.
- **Sidecar tests**: the terminal event carries the state for Claude and
not for Codex. A full ledger reports unknown.
- **Rust** `tests/acpx_quiet_turn_continuation.rs` (4) and
`tests/support/acpx_quiet_turn_restart.rs` (7):
- a quiet Claude turn continues in the same process, and a replay of a
settled call id is refused;
  - live, unknown and malformed reports restart;
- background work that appears after the terminal event restarts at turn
start;
  - failed, cancelled and interrupted turns restart;
  - Codex with a quiet report restarts;
  - an active goal restarts;
  - 4,096 kept turns, then the next turn restarts;
- park after a kept process still prepares a new process, and the next
run attaches to it;
  - a Runner restart after a kept process resumes in a new process.
- The existing restart tests did not change. The fake sidecar sends no
report by default, so they are the tests for the unknown case.

Live run with a real Claude account, not in CI: the real ACP runtime
(ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code
client), a real tool bridge, and the real Anthropic API with model
`claude-sonnet-5-5`, on the code at the top of this stack. A turn that
started one background shell reported `{ live: 1 }`. A turn that started
two reported `{ live: 2 }`, then `{ live: 1 }` and `{ live: 0 }` as each
shell finished. A turn in which Claude waited for its background shell
in the foreground reported the unknown state.

Not run: server integration tests (the fake sidecar there sends no
report, so their process counts do not change), Linux, and a live run
through the real sidecar process and runnerd.

## Risks

- **All Claude sessions now negotiate the `asyncTasks` capability.**
`claude-agent-acp` then sends `async_task_*` updates and adds a marker
to a background Bash update. ACPX drops the updates. No other behavior
of `claude-agent-acp` changes, from a read of its code.
- **Park after a kept process pays the restart again**, as it did before
#20190. With real Claude that restart was about 1.5 s.
- **A small window remains** between the snapshot and the prompt of the
next turn. Claude can start a cycle of its own in that window only
without a background task, because a process that had a background task
is not kept.
- **Activity between prompts is found only when no client request is in
flight.** A request that never gets a response would hide such activity.
- **Background watchers cannot be told apart from other tasks**, so each
task that is not a shell command makes the state unknown.
- **Each kept turn adds one `session.snapshot` request.**
- A multi-turn run with real Claude now uses one provider process. This
path has tests with the fake sidecar and with the real ACP runtime. It
has no test with the real sidecar host.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, and ran the TypeScript and Rust verification
again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the signals
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 12:54:43 -08:00
DottaandClaude Opus 5.5 969cf246ae perf(run-events): keep each native source's cursor on the run row (#21435)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Each native source event append counted all retained events of its
source (`count(*)`, `max(source_seq)`) to compute the acknowledgement
cursor
> - So the cost of each append grew with the length of the run, and long
runs made every text delta slower
> - This pull request keeps the gap-free cursor of each source on the
run row and moves it in the same UPDATE that reserves the event sequence
> - The benefit is that an in-order append stops reading other events,
so its cost does not grow with run length

## Linked Issues or Issue Description

Refs #21131 (earlier reduction of statements per event). Item 1b of the
agent scaling plan in #21418.

**What existing behavior does this improve?**
The append of a native source event (`appendNativeSourceEvent` in
`server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE
that counted every event of the source on every append.

**Subsystem affected**
Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`).

**Current behavior**
On a run that already holds 50,000 events, one in-order append takes
about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it
takes about 3.3 ms.

**Proposed behavior**
The append reads the stored cursor in the run lock statement that it
already sends. An in-order event moves the cursor and does not count
events. On the same 50,000-event run an append takes about 1.7–3.3 ms
(p50).

**Reason and benefit**
Text deltas are about 85% of the server's statements in the load model.
Their cost must not grow with run length.

**Breaking changes**
None. The new column is nullable. The previous release ignores it.

## What Changed

- Add the nullable column `heartbeat_runs.native_source_cursors`
(`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD
COLUMN` only and classifies as `online_compatible`.
- The run lock read of a native append also reads the cursor of its
source.
- When the event sequence is at most cursor + 1, the append statement
has no `retained` CTE. It moves the cursor in the UPDATE that reserves
the sequence, and the receipt uses the cursor.
- Every other case (no cursor, a gap, or a cursor behind the stored
events) runs the earlier recount. In the same UPDATE it stores the
cursor when the source has no gap, and removes it when the source has a
gap. So the event that fills a gap recounts.
- The cursor is a lower bound. A writer that does not keep it (the
previous release during a deploy overlap, or
`NativeRunCoordinatorStore`) can only make it lag. The next out-of-order
event then recounts and corrects it.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests).
- Red/green: before the change, 3 tests failed: the two new tests and
the updated lock-read statement test. After the change, all pass.
- New: "keeps the source cursor on the run so an in-order append does
not count the run's events" checks that the append statement has no
`count(` and that a replay returns the cursor.
- New: "recounts a source whose run has no cursor or a cursor behind its
stored events" covers rows saved before the change, rows from a writer
that does not keep the cursor, a gap, and the event that fills the gap.
- Focused related tests: `paperclip-control-plane-port.test.ts`,
`native-observer-replay`, `native-event-boundary-golden`,
`chat-native-progress-work`, `live-recovery-work`,
`native-interaction-bridge`.
- `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2
(broader CI coverage required) because of local benchmark scripts that
are not in this pull request. Required CI supplies the broad coverage.
- Append cost on a run with many events (local embedded PostgreSQL,
1,000 in-order appends, busy machine, load average about 20–40):

  | Events already in run | Before p50 / p90 | After p50 / p90 |
  | --- | --- | --- |
  | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms |
  | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms |
  | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) |

- Statements per event do not change: BEGIN, run lock read, one append
statement, COMMIT (4). The append statement no longer scans the source's
events.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`): the machine was too busy for a stable result.
Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After:
13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of
20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second
window gives only 400 events per run, so it cannot show this change. The
long-run figures above are the evidence.

## Risks

- Low. The column is nullable and the previous release does not read it.
Runs that are in flight during a deploy have no cursor. Their first
append recounts once and stores the cursor.
- A stale cursor can only make the acknowledgement lag, never pass the
stored events. Event rows are deleted only with their company or agent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:08 -06:00
DottaandClaude Fable 5.2 0199c963e2 feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)

## Linked Issues or Issue Description

No public issue exists. Enhancement:

**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.

**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.

**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.

**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".

**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.

**Breaking changes**
None. The five auth modes and their request payloads do not change.

**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.

## What Changed

- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.

Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.

## Verification

- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.

## Risks

- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:35:49 -06:00
ef2a49b6df fix(daytona): bound the duplex channel close so a stalled kill cannot retire the worker (#21471)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud agent runs use the Daytona sandbox provider plugin. One shared
plugin worker process serves all Daytona calls on a server.
> - The host closes a duplex channel through the worker. The host waits
10 seconds for the close acknowledgement. When the acknowledgement is
late, the host retires the whole worker with SIGKILL.
> - The worker close waits for the remote PTY kill before it releases
the socket. The Daytona SDK sends that kill as a toolbox request with a
24-hour default timeout. A stalled sandbox can hold it open for longer
than 10 seconds.
> - Then the host kills the shared worker. Every other call in flight on
that worker fails with `Worker process exited (code=null,
signal=SIGKILL)`. This includes other runs' workspace preparation and
restore.
> - This pull request limits the wait for the remote kill to 5 seconds.
Then the worker releases the socket and acknowledges the close.
> - The benefit is that one slow sandbox no longer stops unrelated runs
on the same server.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging server, the host logged `duplex channel close not
acknowledged; retiring worker` six times in about 90 minutes. Each time,
the Daytona worker exited with SIGKILL. Each time, all in-flight calls
on that worker failed. Runs in workspace preparation failed with
`setup_failed` and `Worker process exited (code=null, signal=SIGKILL)`.
Several runs failed together in each burst.

**Expected behavior**
A slow close of one duplex channel does not stop calls that belong to
other channels or other runs.

**Steps to reproduce**
1. Open a duplex channel on a Daytona lease.
2. Make the Daytona PTY kill request stall (the SDK default request
timeout is 24 hours).
3. Close the channel from the host. The worker does not acknowledge
within 10 seconds.
4. The host retires the worker with SIGKILL. All other pending calls
fail.

**Relevant logs or output**
```
info duplex channel close not acknowledged; retiring worker
info retired worker process exited  code=null signal=SIGKILL willRestart=true
JsonRpcCallError: Worker process exited (code=null, signal=SIGKILL)
```

**Additional context**
The login PTY close does not have this problem. It only does the local
socket disconnect. A kill that rejects already gets a close
acknowledgement, because the session close ignores kill errors. This
change gives a slow kill the same result.

No related public PR exists. I searched open and closed PRs for this
failure.

## What Changed

- `duplex-command-stream.ts`: `close()` waits for the remote `kill()`
for at most `DUPLEX_CHANNEL_CLOSE_KILL_TIMEOUT_MS` (5 seconds). Then it
always releases the socket with `disconnect()`. The kill continues in
the background. A kill that answers in time keeps the old order: kill,
then disconnect, then the acknowledgement.
- `duplex-close-deadline.test.ts` (new): regression tests at the session
close and at the worker `onDuplexChannelClose` boundary, and a check
that the deadline stays well inside the host deadline.

## Verification

- Red before the fix: with the old `duplex-command-stream.ts`,
`acknowledges a bound close inside the host deadline when the remote
kill never settles` fails (no acknowledgement after 9 seconds), and the
session close test times out.
- Green after the fix: `pnpm --dir
packages/plugins/sandbox-providers/daytona exec vitest run --config
vitest.config.ts src/duplex-close-deadline.test.ts
src/duplex-command-stream.test.ts src/plugin.test.ts` passes 453 of 453.
This includes the existing tests `joins pending teardown for concurrent
closes while all channel authority stays revoked` and `does not turn a
rejected retired close into an acknowledgement`.
- The safety case `still waits for a kill that answers inside the
deadline before it releases the socket` passes before and after the fix.
- `tsc --noEmit` for the Daytona plugin passes.
- In the full Daytona plugin suite, two GNU tar tests in
`file-sync.test.ts` fail in my sandbox. They also fail on unchanged
`master` in the same sandbox, so they are environment failures. CI
covers them.
- I did not run a live staging workflow. After the deploy, the staging
logs must show no new `duplex channel close not acknowledged; retiring
worker` lines from slow kills.

## Risks

- Low risk. The change is in one Daytona plugin function.
- After a kill timeout, the gateway process can stay alive in the
sandbox until the background kill completes or the lease teardown stops
the sandbox. The socket is closed, so no bytes can move on the channel.
The route stays retired, so the host route cannot open again. This is
the same state as when a kill rejects today.
- The host contract does not change. A malformed or mismatched
acknowledgement still retires the worker.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
2026-11-13 12:17:05 -08:00
7773b52f05 fix(codex): preserve sandbox certificate trust in Codex shell commands (#20364)
<!-- Written in ASD-STE100 style: short sentences, active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner and the Codex adapters run agent shell commands
inside a sandboxed Codex session with a strict environment allowlist
> - The controller builds `shell_environment_policy.include_only` before
the sandbox injects certificate-trust variables such as `SSL_CERT_FILE`
and `GIT_SSL_CAINFO`
> - The allowlist drops the injected variables, so managed `git` and
`gh` commands cannot verify HTTPS against a runtime-provided CA
> - This pull request allows the certificate-trust variable names
through every environment policy stage, from the Rust process supervisor
to the shell allowlist
> - The benefit is that managed `git` and `gh` work again in sandboxes
that require a runtime CA, while provider-credential isolation and the
bounded allowlist stay in place

## Linked Issues or Issue Description

- Fixes: #20019
- Refs #20038 — community pull request for the same defect. This change
reviews the approach independently. The commit credits the community
author as co-author.

## What Changed

- Add `CERTIFICATE_ENVIRONMENT_KEYS` (`SSL_CERT_FILE`, `SSL_CERT_DIR`,
`GIT_SSL_CAINFO`, `NODE_EXTRA_CA_CERTS`, `DENO_CERT`,
`REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`) to
`packages/paperclip-runner/src/configured-environment.ts`
- Native Codex shell policy (`codex-security-config.ts`): always put the
certificate names in `include_only`; set inheritance to `"all"` with
`ignore_default_excludes=true`, bounded by the explicit allowlist
- Pass the certificate names through the app-server environment filter,
the ACPX spawn environment filter, the ACPX runtime sandbox
`include_only`, the runner platform environment keys, and the Rust
process supervisor base key list
- Legacy and ACPX-engine Codex shell allowlists
(`packages/adapter-utils/src/codex-shell-environment.ts`): always
include the certificate names
- Add a synthetic private-CA HTTPS journey test: a real `git ls-remote`
fails TLS verification without the injected CA and succeeds with it,
through the policy-filtered child environment
- Document the contract in `doc/sandbox-shell-environment.md` and link
it from `doc/DEVELOPING.md`

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/codex/codex-security-config.test.ts
src/configured-environment.test.ts
src/drivers/codex/codex-shell-certificate-trust.test.ts
src/drivers/codex/codex-app-server-driver.context-control.test.ts
src/drivers/acpx/runtime-sandbox.test.ts` — 104 tests pass
- `pnpm --filter @paperclipai/adapter-utils exec vitest run
src/codex-shell-environment.test.ts src/acpx-engine/execute.test.ts` —
270 tests pass
- `pnpm --filter @paperclipai/adapter-codex-local exec vitest run
src/server/execute.test.ts` — 10 tests pass
- Codex driver lifecycle, conformance, recovery, and boundaries suites
pass (103 tests); environment-builder suites pass (307 tests)
- Red proof: with only the source fix reverted, the new regressions
fail. The journey test shows `GIT_SSL_CAINFO` is undefined in the shell
environment, and `git` fails TLS verification against the private-CA
fixture
- `cargo check` passes for `paperclip-runner-core`; `rustfmt --check`
passes for the edited Rust file
- The journey test needs `git` and `openssl`; it skips when either tool
is absent

## Risks

- The native shell policy now always sets
`shell_environment_policy.inherit="all"` with
`ignore_default_excludes=true`. Before, it set `"none"` when no
credentials or task keys were projected. The explicit `include_only`
allowlist still bounds every inherited name, so no new secret names can
cross
- The allowlist grows by seven non-secret certificate-trust names. Their
values point at CA files. The sandbox filesystem policy still controls
access to those files; this change grants no filesystem access
- No persisted run, session, or launch identity changes shape. Sessions
saved by the previous release resume unchanged; the new policy applies
when the next launch builds its arguments
- Provider credentials keep their existing exclusions at every stage

## Model Used

- Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended
thinking, agentic tool use, run through Claude Code. The implementation
was written and reviewed independently against the community design in
#20038; the community author is credited as co-author in the commit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending: checks run after this
PR opens)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: everlastingcrown <everlastingcrown@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-11-13 14:13:01 -06:00
Nicky LeachandClaude Fable 5.2 7866e9ef05 fix(acpx): admit a Claude tool call only for a tool use its turn announced (#21278)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar. Claude
calls Paperclip tools through a Runner-owned MCP endpoint, the tool
bridge.
> - The bridge identifies a call by its JSON-RPC request id. That id
does not say which turn sent the call. The sidecar binds a call to the
turn that is active when the call arrives.
> - Because of this, the Runner stops and starts the Claude process at
each turn boundary. A late call from an earlier turn then cannot run in
a later turn. This restart also stops all background work that Claude
started.
> - Claude Code already sends the missing identity. Each MCP
`tools/call` has `_meta["claudecode/toolUseId"]`. It is the id of the
ACP `tool_call` that the provider announces first, inside one turn.
> - This pull request uses that id. The sidecar admits a Claude tool
call only if the active turn announced its tool use.
> - The benefit is that a call from an earlier turn is refused by its
identity. This is the first step to keep the Claude process alive
between turns. This pull request does not change the restart.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #19912 (the design for turn-scoped ACPX MCP authority), #12419 (the
restart at turn start) and #20190 (the restart at turn end).

This pull request is the bottom of a stack. #21274 is merged, so the
base branch is `master`. It uses no code from #21274. The stack order
lets a later pull request remove the direct gateway path after both.

**What existing behavior does this improve?**

How the Runner decides which turn a Claude tool call belongs to.

**Subsystem affected**

Paperclip Runner: the tool bridge and the ACPX sidecar, for the Claude
provider only.

**Current behavior**

- The bridge keys a call by the JSON-RPC request id
(`runner-tool-bridge.ts`).
- The sidecar gives the call to the active turn (`waitForTool` in
`acpx-runtime-sidecar.ts`).
- After `session/cancel`, Claude Code does not stop an MCP request that
is in flight. A test run showed two calls from two turns live at the
same time on one MCP session. Request ids cannot tell them apart.

**Proposed behavior**

- The bridge can read a provider tool-use id from `params._meta` and
give it to the handler. It keys replays by that id.
- For Claude, the sidecar admits a call only when the turn event stream
of the active turn contains the ACP `tool_call` with the same id, and no
earlier call used that id.
- The announcement and the call use different channels. If the call
arrives first, the sidecar waits for the announcement for a maximum of 5
seconds. The wait stops when the turn ends or the call is cancelled.
- A call with no id, an id from an earlier turn, an id that was never
announced, or an id that was used before gets a tool error. There is no
fallback.
- The call id that the sidecar sends to runnerd is `tool-use:<id>` for
Claude.

**Reason and benefit**

- A late call from an earlier turn cannot run in a later turn. The
identity of the call prevents this, not only the process restart.
- A later change can stop the restart between turns.

## What Changed

- `packages/paperclip-runner/src/drivers/runner-tool-bridge.ts`
- New option `providerToolUseIdMetaKey`. Without it the bridge does not
change.
- With it, the bridge reads `params._meta[key]`. It refuses a value that
is not a bounded string. It gives a valid value to the handler as
`providerToolUseId`.
- The replay key is the tool-use id. A retry with a new request id
replays the first result. The same id with different arguments is
refused.
  - `notifications/cancelled` still finds the call by its request id.
-
`packages/paperclip-runner/src/drivers/acpx/claude-tool-use-admission.ts`
(new): `ClaudeToolUseAdmission`.
- `observe` records an id only from an event with `type === "tool_call"`
and `tag === "tool_call"`. A `tool_call_update` does not count, so a
late update about an old tool use cannot admit it.
- `admit` applies the rule above. An id is used when it is admitted.
Limits: 4,096 announcements for each turn and 8,192 admitted ids. At a
limit, new calls are refused.
- `packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts`
- The sidecar creates the admission object only for `agent === "claude"`
and sets the bridge option only for Claude.
- `pumpTurn` gives each turn event to `observe`. `waitForTool` calls
`admit` and then checks the turn and run again.
- The internal `get_task_context` call for a completion continuation
does not go through admission.
- The Claude call id does not use the per-process namespace, because a
tool-use id is unique.
- No change to Rust, the sidecar contract, the `claude-agent-acp` patch,
the server, or restart, park and turn lifecycle code. Other ACPX agents
and Cursor Cloud use the same code path as before.

Why 5 seconds: the measured time between the announcement and the call
is 1.1 ms to 14.7 ms. The limit must also include queue and event-loop
delay. 5 seconds is far above that, and a call that cannot be attributed
still gets a fast answer.

## Verification

Commands, from `packages/paperclip-runner`:

- `npx vitest run
src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts
src/drivers/acpx/claude-tool-use-admission.test.ts
src/drivers/runner-tool-bridge.test.ts
src/drivers/runner-tool-bridge.claude-tool-use.test.ts
src/drivers/runner-tool-bridge.claude-schema.test.ts
src/cli/acpx-runtime-sidecar.test.ts
src/drivers/acpx/acpx-runtime-adapter.test.ts
src/drivers/cursor-cloud/cursor-cloud-driver.test.ts`: `8 passed (8)`
files, `384 passed` tests.
- `npx tsc --noEmit -p tsconfig.json`: no errors.

New tests:

- **Real ACP runtime** (`acpx-runtime-adapter.claude-tool-use.test.ts`).
It uses the real `openCodexAcpxRuntime`, ACPX 0.22, the pinned and
patched `claude-agent-acp`, the pinned Claude Code client and a real
tool bridge, against a local fake Anthropic API.
- Turn 1: two parallel tool uses are admitted as `tool-use:<id>`. The
model receives both results. The turn completes.
- Turn 2 in the same session: a replay of a turn 1 id returns the cached
result and does not dispatch again. The same id with different arguments
is refused. `admit` refuses the old id.
- In five runs each call arrived 6.9 ms to 13.2 ms after its
announcement. No call waited.
- **Qualification** (`runner-tool-bridge.claude-tool-use.test.ts`). The
pinned Claude Code client sends `_meta["claudecode/toolUseId"]` on each
`tools/call`, with the id that the API issued. This test must pass
before a Claude Code version change.
- **Admission unit tests** (19): announced then called, called then
announced, never announced, turn end during the wait, cancel, an id from
the previous turn, reuse, two concurrent calls with one id, two turns
with live calls, a subagent call, a bad id.
- **Sidecar process tests** (2), with a fake host, for Claude and for
Codex. The Codex run shows no change.
- **Bridge unit tests** (5).

Mutation checks:

- Change the announcement predicate so that it matches nothing: the real
ACP test and one unit test fail.
- Remove the bridge option in the real ACP test: the test fails.
- Count only `tool_call_update`: the unit test "counts only an opening
ACP tool_call event as an announcement" fails.

Live run with a real Claude account, not in CI: the real ACP runtime
(ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code
client), a real tool bridge, and the real Anthropic API with model
`claude-sonnet-5-5`, on the code at the top of this stack. In three
sessions, 8 Paperclip tool calls were admitted and none was refused.
This includes calls in a prompt turn, in a cycle that Claude started
itself, and in a later turn of the same process. Each call arrived 0.8 s
to 1.7 s after its announcement, so no call waited for a late
announcement.

Not run: a turn through the real sidecar process with the real host.
That needs a qualified provider installation, the runtime sandbox and
the provider lifetime lease. Not run: the Linux builds of the pinned
client and a test under load. Not run: `pnpm verify` and the full suite.
CI supplies that coverage.

## Risks

- **This changes each Paperclip tool call that Claude makes.** If the
announcement does not reach the turn event stream in production, the
sidecar refuses the call after 5 seconds. The real ACP test covers the
stream of ACPX 0.22 with the pinned provider. It does not cover the real
sidecar host.
- **The 5 second limit is not tested under load.** A slow host can delay
the announcement. The call is then refused, and Claude sees a tool
error.
- **A Claude Code version that stops sending the id breaks all Claude
tool calls.** The qualification test finds this before the version
changes.
- **The admitted-id ledger is in sidecar memory.** Today the sidecar
restarts at each turn boundary, so the ledger is small. The limit of
8,192 ids refuses new calls when it is full.
- **Subagent calls and calls from a cycle that Claude starts itself**
are covered by unit tests and fake-host tests, not by the real ACP
stream.
- **An admitted call that is in flight when its turn ends** is refused
by the existing turn cleanup. Claude Code can leave the HTTP request
open. The bridge stops it only on its own cancel or close, as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, asked for the real ACP runtime test, and ran
the verification and a mutation check again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 11:56:42 -08:00
DottaandClaude Opus 5.5 38821e061f perf(runner): serialise only changed journal records for a checkpoint (#21437)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running agent, and its main thread
limits how many agents can run at the same time
> - Each agent has a controller journal. The controller saves a
checkpoint of this journal after each control event and before each
frame it sends
> - Each checkpoint converted every command and every event of the
journal to JSON again, also when only one record changed. The cost grew
with the journal length (up to 500 commands and 4,096 events)
> - This pull request keeps the JSON of each record and uses it again
while the record does not change
> - The benefit is that a checkpoint converts only the records that
changed, so the main thread does less work for each agent

## Linked Issues or Issue Description

Refs #21411 (this change sits on top of its checkpoint queue).

**What existing behavior does this improve?**
The controller journal checkpoint in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.
Each flush calls `captureCheckpoint`, which runs `JSON.stringify` on
every command and every recovery event of the journal.

**Current behavior**
A flush after one new event converts the full journal to JSON. With
4,096 events, one flush made about 5 MB of new JSON strings and used
about 14 ms of CPU on the event loop. A hosted CPU profile with about 50
agents showed about 3% of the main thread in this path.

**Proposed behavior**
A flush converts only the records that changed since the last flush. The
other records use the JSON that the controller made before. The saved
records are byte-identical to the records of the full conversion.

**Reason and benefit**
The target is 1,000 agents in one controller process. Checkpoint CPU
must not grow with journal length. After this change, one flush with
4,096 events makes about 2.5 KB of new JSON and uses about 2 ms of CPU.

## What Changed

- Add `CheckpointRecordCache` to `durable-prp-control-plane.ts`. It
keeps the JSON and the part name of each command and event record, keyed
by the record object. It uses the JSON again only when the record has
the same keys, in the same order, with the same values (`===`). Commands
change some fields in place (`status`, `result`, `controllerSeq`), and
this comparison finds these changes.
- `captureCheckpoint` accepts the cache as an optional fourth argument.
Only `setCheckpointPersistence` and `flushCheckpoint` pass it. All other
callers (decode, legacy import, bootstrap inspection) do the full
conversion as before.
- `checkpointChanges` uses loops instead of array copies. Records that
did not change keep the same string, so the comparison is by reference.
- `queueCommand` stores a copy of the command payload
(`structuredClone`). The journal must own its nested objects, because
the cache compares nested objects by reference.
- `durableRecoveryInternals` exposes `captureCheckpoint` and
`createCheckpointRecordCache` for tests.
- Regions changed: `checkpointChanges` and the new cache above
`captureCheckpoint`; `captureCheckpoint`; `durableRecoveryInternals`;
one field of `DurablePrpControlPlane`; the `captureCheckpoint` calls in
`setCheckpointPersistence` and `flushCheckpoint`; the `payload` field in
`queueCommand`. The `#event` path is not changed.
- No change to what a checkpoint saves, when a flush is awaited, or the
order of event commit, acknowledgement and checkpoint. The "at most one
waiting checkpoint" behavior from #21411 is not changed.

## Verification

- New `durable-prp-checkpoint-records.test.ts`:
- A journal with 499 commands and 4,000 events gets one new command. The
test counts `JSON.stringify` calls on journal records during the flush.
Before: 4,499. After: 0 calls on existing records, at most 2 in total
(the size check of the new command and its checkpoint record). The delta
contains only `command:500`. The test then restores a new controller
from the checkpoint and compares its records with a full conversion
without the cache. I saw this test fail on `origin/master`.
- A property-style test with 5 seeds and 600 random steps each: add,
settle and remove commands (with `controllerSeq` renumbering), add,
replay and evict events, streaming and usage events, `structuredClone`
and spread copies of the state, header counters, the same value written
again, and a key-order change. After each step, the records with the
cache must be byte-identical to the records without the cache, in the
same order. I checked that this test fails when the field comparison is
removed (cache by object only).
- Existing `durable-prp-control-plane.test.ts` passes. On this loaded
machine, some 5-second timing tests time out with and without this
change (they also time out on `origin/master`), and pass when I run them
alone.
- Benchmark: `PAPERCLIP_CHECKPOINT_FLUSH_BENCHMARK=1 pnpm exec vitest
run src/control-plane/durable-prp-checkpoint.benchmark.test.ts` (from
`packages/paperclip-runner`). Each case commits one new event and
flushes with a no-op save, 200 times. CPU time is noisy (load average
about 40), the byte and record counts are exact.

| Journal | CPU per flush, before | CPU per flush, after | JSON bytes
per flush, before | after | Records converted per flush, before | after
|
| --- | --- | --- | --- | --- | --- | --- |
| 10 commands, 10 events | 207 µs | 64–80 µs | 20,975 | 2,465 | 20 | 1 |
| 500 commands, 10 events | 1,874 µs | 505–511 µs | 375,068 | 2,466 |
510 | 1 |
| 10 commands, 4,096 events | 14,013 µs | 1,655–2,343 µs | 5,174,786 |
2,473 | 4,106 | 1 |

- `pnpm check:module-boundaries` passes. `tsc -p
packages/paperclip-runner/tsconfig.json --noEmit` passes.

## Risks

- The cache assumes that nested `payload`, `result` and `envelope`
objects of journal records are not edited in place after they enter the
journal. I checked the module: results are stored with
`structuredClone`, envelopes with `structuredClone`, and events are
replaced, not edited. This change also copies the payload in
`queueCommand`. If a later change edits a nested object in place, the
checkpoint could keep old JSON. The property test shows top-level edits
are found.
- Some per-record work is still proportional to the journal (a Map of
part names and a field comparison for each record). It is about 0.4 µs
for each record. Removing it needs dirty tracking in every state
mutator, which this change does not do.
- No change to the saved format, so old and new servers read the same
checkpoints during a deploy overlap.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:21:51 -06:00
commitperclip[bot] 80f7989f4b fix: Wait for the SSH export pipeline to drain before importing the bundle (#20708)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

## Thinking Path

- Issue #16567 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: SSH process closure ended the
destination stream while the progress Transform still held buffered
bytes. Transfer success now requires both pipeline completion and SSH
completion.
- This pull request: Wait for the SSH export pipeline to drain before
importing the bundle

## What Changed

- `packages/adapter-utils/src/ssh.ts`: Use Node's pipeline to drain the
stream chain, propagate stream errors, and release streams on failure
while preserving SSH stderr errors.
- `packages/adapter-utils/src/ssh-export-stream.test.ts`: Reproduced
truncation and confirmed the fix passes with and without progress. Added
failure cleanup cases; their final run and typecheck have not returned,
and live SSH fixtures skipped because ssh is unavailable.

## Verification

- Reproducing test(s), added in this PR:
`packages/adapter-utils/src/ssh-export-stream.test.ts`,
`packages/adapter-utils/src/ssh-export.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- 3 independent attempts wrote 3 reproductions; this fix passes every
one of them.
- The repository's compile command passes with the fix applied.
- 10 existing test file(s) near the change failed, and fail on the base
too, so they cannot show whether the change breaks them.
- The full test suite passes except for the 17 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.

## Model Used

- codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code;
OpenAI, via the Codex CLI), writing and checking the change in an
isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #16567

#16567 also mentions, without asking this fix for it:
- Add git bundle verify or a remote size check to provide a clearer
diagnostic for short files. (The report calls this 'A cheap extra guard
would be git bundle verify (or a size check sent by the remote script)
before git fetch'; it is an optional additional safeguard. Empty or
malformed output that ends normally still reaches the existing git fetch
validation at packages/adapter-utils/src/ssh.)

Other pull requests that name #16567:
- #16613 (open): fix: prevent truncated SSH Git exports

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=16567 base=5ab98b006359
report=6118c07135df7c3a -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:59:04 -06:00
commitperclip[bot] f4159e25e4 fix: Remove the unused OpenCode remote config guard and document host staging (#20045)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

> [!WARNING]
> This change crosses a trust boundary. See Risks.

## Thinking Path

- Issue #17372 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: The helper's unused
targetIsRemote guard contradicted both callers, which intentionally
stage configuration on the host and transfer it to remote targets as
xdgConfig.
- This pull request: Remove the unused OpenCode remote config guard and
document host staging

## What Changed

- `packages/adapters/opencode-local/src/server/runtime-config.ts`:
Remove the unused option and contradictory guard while preserving both
callers' staging behavior.
- `packages/adapters/opencode-local/src/server/runtime-config.test.ts`:
Add a regression for isolated staging, generated configuration, cleanup,
and operator config preservation. Confirmed failure before the fix; all
50 focused tests pass afterward. Typecheck was started but has not
returned a result.
- `packages/adapters/opencode-local/README.md`: Document host staging
for local and remote execution, remote asset transfer, path replacement,
and host cleanup.

## Verification

- Reproducing test(s), added in this PR:
`packages/adapters/opencode-local/src/server/runtime-config.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- 3 independent attempts wrote 3 reproductions; this fix passes every
one of them.
- The repository's compile command passes with the fix applied.
- 9 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.
- It crosses a trust boundary. Check each crossing against what the
issue asked for, and against who asked:
- validation
(packages/adapters/opencode-local/src/server/runtime-config.ts:107):
Removes the targetIsRemote early-return check that prevented host
configuration staging when explicitly set, although neither production
caller supplies it.

## Model Used

- codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code;
OpenAI, via the Codex CLI), writing and checking the change in an
isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #17372

#17372 also mentions, without asking this fix for it:
- The remote per-run directory is never removed. (The report explicitly
says: “Related but distinct: #14527 covers the remote per-run directory
never being removed.”)

A search of paperclipai/paperclip's open and closed pull requests found
no other pull request that names #17372.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=17372 base=ca34bed25797
report=0010c91463516eaf -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:58:03 -06:00
Dotta 1e33415f67 test(runner): run Kimi layout suites once in prep (#20487)
<!-- All text in Simplified Technical English. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner package has node:test suites that verify the pinned Kimi
Code native runtime layout.
> - `test:typescript:prep` selects each Kimi layout suite directly. The
same command also selects the `build-verified-provider-entrypoints`
aggregator. The aggregator imported the same four Kimi suites.
> - One command therefore registered every Kimi layout test twice.
Commit b866fa4bd (#18028) added both routes in the same change. The
duplicate has no second contract: both routes run the same files,
assertions, and fixtures in the same required run.
> - This pull request removes the aggregator imports. The direct prep
selection stays. Each Kimi suite now runs once per command.
> - The benefit is less test work and less CI time in each prep run,
with the same assertions.

## Linked Issues or Issue Description

No public issue exists. Problem description:

**What existing behavior does this improve?**

The `test:typescript:prep` command in
`packages/paperclip-runner/package.json`. It selects test files for
`pnpm test:typescript` and `check:protocol` (release verification).

**Current behavior**

The command passes `build-verified-provider-entrypoints.test.mjs` and
the four Kimi suites to one `node --test` run. The aggregator file also
imports the four Kimi suites. Each Kimi test file then registers its
tests twice: once inside the aggregator process, and once in its own
process. Local measurement on the exact prep selection: 81 test
registrations, 71 pass, 10 skip, about 6.6 seconds.

**Proposed behavior**

The aggregator keeps its own two tests and its imports of suites that
prep does not select (`provision-pi-package`, the four Antigravity
suites). The four Kimi suites run only through their direct prep
selection. Same selection of the aggregator, same files, same
assertions. Local measurement after the change: 62 test registrations,
56 pass, 6 skip, 0 fail, about 4.6 seconds. The removed 19 registrations
are exact duplicates; each Kimi test still runs once.

**Reason and benefit**

Double registration wastes CI and developer time on each prep run and
makes test counts misleading. One run of each suite keeps the same
regression proof.

**Breaking changes**

None. No production code changes. The suites and their assertions are
unchanged.

## What Changed

- Removed the four Kimi suite imports from
`packages/paperclip-runner/scripts/build-verified-provider-entrypoints.test.mjs`
(`materialize-kimi-distribution`, `provision-kimi-package`,
`patch-kimi-runtime`, `kimi-metadata-probe`).
- Added a comment that says the Kimi suites are selected directly by
`test:typescript:prep`, so this aggregator must not import them again.
- No changes to `package.json`, routing, or the suites themselves.

## Verification

- Checked the routing: `test:typescript:prep` in
`packages/paperclip-runner/package.json` lists all four Kimi suites
directly, so each suite stays selected with multiplicity one after this
change. The aggregator is only run through that same command; no
workflow or script runs it alone.
- Baseline (master `233f61564`): `node --test` with the exact prep
selection of the aggregator plus the four Kimi suites -> 81 tests, 71
pass, 0 fail, 10 skipped, 6.6 s.
- After the change (this head): same command -> 62 tests, 56 pass, 0
fail, 6 skipped, 4.6 s. 19 exact duplicate registrations removed; the
Kimi suites still run and pass once each.
- Timing is a single local sample; treat the speed difference as
indicative, not proven. The structural change (19 duplicate
registrations removed) is exact.
- Not run: full `pnpm -r typecheck`, full workspace test suite, Rust
checks. A filtered dependency install was used because the workspace
disk is 10 GB and a full workspace install exceeds it. Required PR CI
supplies the typecheck, build, and vitest coverage for this head.

## Risks

Low risk. The aggregator still loads and its own two tests pass. If a
future lane runs the aggregator alone, the Kimi suites would not run
there; today no such lane exists, and the prep command still selects
them directly.

## Model Used

- Provider: Z.ai, model GLM (assisted by Paperclip's Test Gardener
agent).
- Exact model ID: `openrouter/z-ai/glm-5.3`.
- Reasoning/thinking mode: standard chain-of-thought; no extra thinking
mode.
- Capabilities: tool use (shell, file edit, GitHub CLI), repo test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
2026-11-13 12:55:31 -06:00
DottaandClaude Opus 5.5 8aaf983151 perf(db): send a transaction's work notifications in one statement at commit (#21436)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server signals queue work with database work signals, and it
sends a PostgreSQL notification for each signal so that another server
can wake during a deploy overlap
> - Each signal sends its own `select pg_notify($1, $2)`, inside the
open transaction, and waits for it
> - On staging this is the most-called statement in the database: 31.8
per second, 15.1 million calls in total against 7.5 million `begin`,
about 3.7 awaits per transaction
> - Every await inside an open transaction holds its locks and its
connection for one more turn of a saturated event loop
> - This pull request collects a transaction's notifications and sends
them in one statement just before commit
> - The benefit is at most one notification round trip per transaction,
with the same payloads delivered at the same commit

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Database work signals send one `pg_notify` statement per signal, inside
the transaction.

**Subsystem affected**
`packages/db` work signals, used by every delivery queue in the server.

**Current behavior**
`signalDatabaseWork` and `notifyDatabaseWorkOnCommit` each run `select
pg_notify(channel, payload)` on the transaction and wait for it.
`notifyIssueRecovery` alone signals 7 queues per call, and a transaction
often calls it several times. A local legacy run (wake, start, 20 output
lines, finish) sends 424 notification statements. Staging measures 31.8
per second, 8.5 `begin` per second.

**Proposed behavior**
A signal inside a transaction records its payload on that transaction.
Before the outermost commit, the transaction sends every distinct
payload in one statement. Payloads and the channel are byte-identical to
today's, and PostgreSQL still delivers them only on commit.

**Reason and benefit**
At most one notification statement per transaction instead of one per
signal. Fewer awaits while a transaction holds its locks and its
connection.

**Breaking changes**
None. The listener protocol and the payload format do not change.

## What Changed

- `packages/db/src/work-signals.ts`: a signal records `(channel,
payload)` on the current transaction level. A released savepoint merges
its records into its parent; a rolled-back savepoint drops them. The
outermost transaction sends all distinct payloads in one statement
(`select pg_notify(channel, payload) from
jsonb_array_elements_text($2::jsonb)`) after its callback returns and
before commit. A transaction that signals nothing sends no extra
statement.
- The record lives on the transaction object only. A retried callback
gets a new transaction object and starts empty.
- `publishDatabaseWorkHint` (outside a transaction) is unchanged and
still sends at once.
- `packages/db/src/work-signals.test.ts`: tests on real PostgreSQL with
a LISTEN client.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/work-signals.test.ts`: 22 pass. 6 of the new tests fail on master.
- New tests (real PostgreSQL, LISTEN client, asserting what is delivered
as well as what is sent):
- Distinct topics in one transaction: one statement at commit. The
payloads delivered are identical to the format on master (`{version,
origin, topic}`), in first-signal order.
  - A transaction that signals nothing: no notification statement.
- Savepoints: a released savepoint two levels deep keeps its topic. A
rolled-back savepoint drops its topic. A topic from a rolled-back
savepoint that the outer level signals again is still delivered.
  - The callback throws: nothing is sent or delivered.
- The commit-time send fails: the transaction fails and rolls back its
write, and nothing is delivered.
  - 500 distinct topics: one statement, all 500 delivered.
  - Sequential, concurrent and retried transactions each start clean.
- Server suites that use notifications pass: `deployment-notifications`,
`run-start-handover`, `heartbeat-run-start-handover`,
`agent-runtime-hooks`, `legacy-controller-lease`,
`heartbeat-run-dispatch-lifecycle` (99 tests). Two of them hit statement
timeouts once on a loaded machine and then passed 41/41 when run alone.
- Before and after, one local legacy run (embedded PostgreSQL, process
adapter, 20 output lines 50 ms apart, uninstrumented pass, alternating
arms):
`TSX_DISABLE_CACHE=1 UNITS=legacy TASKS=50 AGENTS=3
NODE_OPTIONS=--expose-gc pnpm --filter @paperclipai/server exec tsx
scripts/query-cost/drive.ts` (harness from #21413)

  | Per legacy run | Before (2 runs) | After (2 runs) |
  | --- | --- | --- |
  | Statements | 883, 883 | 474, 491 |
  | `pg_notify` statements | 424 | 21 (one per signalling transaction) |
  | Notification round trips per signalling transaction | about 20 | 1 |
  | Transactions (BEGIN plus COMMIT) | 72 | 72 |
  | Allocated | 81.1, 79.3 MB | 76.3, 74.3 MB |
  | Wall time | 5.0, 5.2 s | 4.1, 7.7 s (noisy: loaded machine) |

Statement counts are exact and stable. Allocation fell by about 5 MB per
run (about 6%). Wall time varies too much on this machine to compare.
- Staging, measured by the coordinator (Neon statistics, 64 seconds):
31.8 `pg_notify` per second, 8.5 `begin` per second. At about 44 KB of
allocation per statement, notifications cost about 1.4 MB/s, about 2.4%
of the 57.6 MB/s. The allocation saving is part of that, about 2%, and
it is not the reason for this change. The reason is round trips inside
open transactions: at most one notification statement per transaction
instead of about 3.7. The per-second effect on staging is an estimate
until it is measured after deploy.

## Risks

- A failure of the notification send now surfaces at commit time, not at
the signal call. No caller catches errors around a signal call and
continues: the only `.catch` near a signal (`notifyBudgetStopsSettled`)
wraps a root-level call that is not in a transaction and already fails
before this change.
- Order within a transaction: notifications are sent in first-signal
order. A savepoint's records join its parent when the savepoint is
released, which matches the order of sequential code. Listeners do not
depend on order: each payload is an independent hint, and PostgreSQL
already removes duplicate payloads within a transaction.
- The notification is sent after the transaction's last statement
instead of in the middle. Delivery still happens only at commit, as
before.

Remaining work, not in this pull request: sending stops being useful
when no other server listens. The listener is open only during a startup
handoff, while this process owns a native execution, or during a
run-start handover, and a server drops its own notifications. A later
change could send only while an overlap is possible. It would have to
prove this interleaving: a new server announces itself, the old server
is still outside its own window, and the old server must start sending
before the new server relies on a hint. Otherwise the new server waits
for its next reconciliation pass.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation describes the send timing)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:39:36 -06:00
DottaandClaude Opus 5.5 d9b977baf2 fix(runner): answer a live Runner's commands while it replays its backlog after a reconnect (#21442)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox is driven by a Runner process. The
server talks to it over PRP and saves the controller state in a
checkpoint.
> - After a release swap, the incoming server reconnects to Runners that
kept working while no server was attached. Each Runner first replays the
events it built up.
> - The server answered commands one at a time and only after every
event queued ahead of them. It wrote one checkpoint and one
acknowledgement per event. A recovery read sent at reconnect therefore
timed out at 30 s behind the replay, although the Runner was alive and
had answered.
> - A tool call that the previous server started and never answered
blocked every later command, and a failed close was replayed forever.
> - This pull request lets read and barrier answers pass the replay,
extends a wait while the replay still commits, acknowledges a replayed
burst once per batch, delivers every pending command at attach, settles
an orphaned tool call at attach, and retries a close whose drain timed
out.
> - The benefit is that a live Runner is reached after a late reconnect
instead of being declared unanswered and then replaced, which lost its
turn.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

Authentication took under 1 s. The identity was recorded 4 to 7 minutes
later, because it is written only after the Runner answers `run.attach`,
`runner.drain` and the thread read. Those answers waited behind the
replayed event backlog. The failure codes on the first attempt were:

- `PRP command session.snapshot timed out` and
`codex_history_read_failed: thread/turns/list read failed` (all 10
`ingress` runs in the sample)
- `PRP command runner.drain timed out (runner connected; pending ahead 2
[semantic_tool.result] oldest 1023s)` on the retry

A separate class, `cleanup_quarantined` (23 failures in 3 hours, no swap
involved), showed "prior session cleanup remains incomplete" on every
retry, because a close that failed once kept replaying the same
rejection.

A live Daytona capture showed the stale `semantic_tool.result`: Codex
waited for the result of an exec tool call that the stopped server
owned. It was not an unacknowledged frame.

**Expected behavior**

A live Runner answers recovery commands within its deadline after a late
reconnect. A tool call left by an earlier controller is settled once
(with its durable receipt, or as interrupted with an unknown outcome,
never executed again). A close that failed only because its journal
drain timed out reaches its real outcome on the next close.

**Steps to reproduce**

1. Start a control plane with a commit callback that takes 50 ms per
event.
2. Connect a Runner that replays 1,000 events, then send
`session.snapshot`.
3. On master the snapshot times out at its 30 s deadline. The new test
`answers a command issued during a 1,000-event replay within its
deadline` does this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

All changes are controller-side TypeScript under
`packages/paperclip-runner/src`.

- `durable-prp-control-plane.ts`: a `command_result` for
`session.snapshot`, `session.goal.get`, `runner.drain` or
`semantic_tool.result` that reports completed or rejected is applied
when it arrives, not after the events queued ahead of it. Failed
results, lifecycle, attachment and turn commands keep their place.
- `durable-prp-control-plane.ts`: the welcome carries every consecutive
pending command (up to 32, within the existing byte bound). A lifecycle
command or `run.attach` that is not first is never batched.
- `durable-prp-control-plane.ts`: an ordinary event inside a replayed
burst is committed at once but checkpointed and acknowledged once per
batch (when the queue drains, when a critical event arrives, or after 64
events). An ACK is still sent only after the checkpoint that contains
its cursor. The owner fence in `flushCheckpoint` is unchanged. Critical
events keep their own checkpoint and ACK.
- `runnerd-codex-transport.ts`: `waitForRunnerCommand` also extends a
deadline while the controller keeps committing the Runner's events. A
Runner with no committed event for 30 s still times out at the original
deadline. A Runner that keeps emitting but never answers times out after
at most 5 more minutes.
- `runnerd-codex-transport.ts`: recovery of a semantic tool input that
an earlier generation dispatched and never answered now also runs when
the controller attaches, not only after startup completes.
`paperclip_finish` and `paperclip_block` still wait for startup.
- `runnerd-codex-transport.ts`: a close that failed only because the
controller journal drain timed out is no longer cached. The next close
drains again with a fresh bound and finishes as settled or as a typed
`NativeSessionCloseUnrecoverableError`.

## Verification

Red on master, green on this branch:

- `durable-prp-control-plane.test.ts`
- `answers a command issued during a 1,000-event replay within its
deadline`
- `keeps a failed command result behind the events the Runner sent
before it`
- `delivers every command pending at attach so an earlier generation's
result cannot hold the recovery probe`
  - `never delivers a lifecycle command ahead of its turn at attach`
- `acknowledges a replayed burst once per batch, never ahead of its
durable checkpoint`
- `still acknowledges a lifecycle event inside a burst before the next
frame`
- `answers a snapshot from a Runner whose provider stays wedged until
its whole replay is acknowledged` (models the Daytona capture: the Codex
Runner stops reading provider output while it holds unacknowledged
events)
- `runnerd-codex-transport-reattach-carried-commands.test.ts`
- `keeps waiting past the deadline while the controller is still
committing the Runner's replayed events`
- `times out a Runner that keeps replaying but never answers once the
replay extension is spent`
- `does not extend a deadline for a Runner whose replay has stopped
advancing`
- `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: `settles an
earlier generation's in-flight tool call at attach even when that attach
never finishes startup`
- `runnerd-detach-durability.test.ts`: new case `close_timeout` in
`joins an output flush before controller retirement permits a successor
writer`

Measured answer latency for a snapshot sent during a replay (50 ms
commit callback): 100 / 1,000 / 10,000 events took 13.95 s / more than
900 s / did not finish on master, and 26 ms / 26 ms / 170 ms with this
branch.

Local runs on this branch: `tsc --noEmit` is clean for
`packages/paperclip-runner` and `server`.
`durable-prp-control-plane.test.ts`: 227 of 227 pass.
`runnerd-codex-transport-reattach-carried-commands`, `-adoption`,
`-orphaned-tool-at-attach` and `runnerd-detach-durability`: all pass.
`runnerd-codex-transport-settlement` and `-deadlines`: the run on this
shared machine (load 50 to 96) had 8 tests fail. A rerun of those two
files alone (load 40 to 72) still had 7 failures: one timing assertion
(`expected 7419 to be less than 3500`) and six settlement cases that
ended with `native_cleanup_maintenance_unproven` after timeouts. The set
of failing cases changed between the two runs. The CI Runner support
suite ran both files on this head and passed, so I rely on CI for them.

Runner content ID: this PR changes only controller-side TypeScript under
`packages/paperclip-runner/src`. There is no Rust, `protocol/` or
`generated/` change. The Runner content ID changes because its hash
covers that source, but existing Runner images work with this
controller. The Daytona cells below ran this controller against an image
built for an earlier content ID.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR and
three others (the reattach-first PR (`fix/reattach-live-runs-first`),
the adopt-live-Runner PR (`fix/adopt-live-runner-on-retry`), and #21362,
which is merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed with
`runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. That chain did not recur. All 6 runs that crossed swap 2 were
adopted on the same Runner process, with no retry and no 900 s timeout.
- Caveat: G5c ran before the reattach-phase timing fix in the
reattach-first PR (`fix/reattach-live-runs-first`). The command fixes in
this PR were present in G5c except the test-only fix for the
wedged-provider test.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection): 0
failed runs at either swap; 11 of 11 survivors at both. Swap 2 was slow
(see the reattach-first PR (`fix/reattach-live-runs-first`)).

## Risks

- Not proven: the Runner-side stall. A Codex Runner stops reading
provider output while no controller is attached (bounded 256-line
channel, provider stdout pipe full). This PR makes the controller
consume the replay quickly, so the stall clears in about 2 to 6 s once
the controller drives it. The durable fix is in the Runner (keep
draining provider output to the durable outbox while disconnected, save
once per ACK batch, run commands ahead of ACKs). That needs a new Runner
image and is not in this PR.
- Not proven: why a connected Runner with an empty outbox sometimes did
not answer for more than 30 s. The G5 capture points to the server
stalling past its own deadline (a `session.open` answered and
acknowledged 8 minutes before the server reported it timed out), not to
the Runner.
- Not done: cleanup takeover for a same-run retry when
`remoteCleanupScope` is absent, and for a predecessor run that shares
the warm sandbox. This PR only stops the replayed close failure.
- Time to reach a wedged Codex Runner still grows with its backlog:
about backlog x per-event commit / database concurrency (40 runs x 1,000
events x 20 ms / 8 is about 100 s).
- Behavior shift: a crash during a batched burst replays at most the
unacknowledged batch (up to 64 events). The Runner still holds those
events, and commit order is unchanged.
- A Runner that keeps emitting but never answers now fails after at most
5 more minutes instead of 30 s.
- The predecessor-retirement PR (`fix/retire-predecessor-session`)
builds on the close path changed here (`retry a close whose controller
journal drain timed out`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code. The fixes were written by Claude agents in the
same session family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:34:31 -06:00
DottaandClaude Opus 5.5 df1b12ad8e fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop

## Linked Issues or Issue Description

**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.

**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.

**Paperclip version or commit**
`master` at f7ddab8e94.

**Steps to reproduce**
1. Run an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, where OpenRouter routes
to Wafer or DigitalOcean.
2. Ask the agent to hand a case to another agent with `handoff_case`.
3. See `handoff_case` arrive with no arguments and get rejected on every
call.

## What Changed

- Add `providerToolSchemaShapeIssues`
(`packages/paperclip-runner/src/contracts/provider-tool-schema-shape.ts`).
It reports a root that is not `type: "object"`, has no `properties`, or
uses `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then`, `else`,
`dependent*` or `$ref`.
- Add inventory tests that run this check on every agent-facing tool:
the live semantic catalog, the semantic action catalog, the Codex tool
specs, `paperclip_finish` and `paperclip_block` (runner), and the
server-only tool definitions (server). Before this change, the runner
test listed 16 entries for 8 tools: `handoff_case`,
`request_human_input`, `read_case`, `update_case`, `create_case`,
`read_workflow`, `update_workflow` and `paperclip_finish`.
- Flatten these 8 schemas. `handoff_case` is now one object with `to`
(optional, `"done"`), and `case` is one object with optional `id` and
`create`. Descriptions on the properties state which fields each variant
needs.
- Add `toolInputVariantError`
(`packages/paperclip-runner/src/contracts/tool-input-variants.ts`). It
checks the former cross-field rules and returns an error that names what
is wrong and how to send the call correctly. For example: `Invalid tool
input for handoff_case: An agent handoff requires toAgentId, phase. To
finish an existing case instead, send to: "done" with
expectedHandoffVersion and expectedIssueStatusVersion.`
- The server runs this check first in
`PaperclipRunnerToolAuthority.execute` and returns `422`. The #21360
guard therefore counts these errors too. Assigned MCP tools and
connector tools are not checked. The runner semantic dispatchers run the
same check after schema validation. The runner MCP bridges run it for
`paperclip_finish`, which never reaches the control plane.
- Add `RepeatedToolInputRejectionGuard`
(`packages/paperclip-runner/src/drivers/repeated-tool-input-rejection.ts`)
and use it in the OpenCode MCP bridge and the runner tool bridge, where
runner-side validation happens. The key is the tool, the canonical
arguments and the error text, separated per MCP session. The second
identical rejection adds the arguments that arrived and the expected
required and optional properties. The third and later identical calls
say that the call was not accepted and that the model must change the
arguments or stop and report the blocker. State is in memory, per
bridge, and capped at 64 entries.
- Regenerate `semantic-tool-contracts.json`,
`semantic-action-catalog.json`, `protocol/manifest.json` and the seeded
eval fixture.
- Add `spec/fixtures/pre-flat-tool-input-schemas.json`: the 8 strict
schemas from the previous release, for the upgrade test.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/contracts
src/catalog src/protocol-actions src/semantic-tools
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts`: all pass.
- `cd server && npx vitest run
src/services/native-runtime/tool-input-variant-admission.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts`: 9
passed.
- Red/green:
- The runner inventory test fails on `master` and lists the 16 entries.
It passes after the change.
- The guard test replays the recorded empty `handoff_case` call 4 times
through the real OpenCode bridge. Without the guard, the second call has
no correction and the test fails. With the guard, the second call lists
`required idempotencyKey, case, instructions; optional to, toAgentId,
phase, ...`, and the third says the call was not accepted.
- Negative controls: changed arguments get a fresh error, and a valid
call still runs once.
- Upgrade safety: `tool-input-variants.test.ts` compiles the previous
release's strict schemas from the fixture. For a corpus of valid and
invalid inputs per tool (27 for `handoff_case`), it checks that "flat
schema accepts and `toolInputVariantError` is null" gives exactly the
same result as the old schema. Each tool has accepted and rejected
inputs.
- Generated outputs: `check:semantic-contracts`,
`check:capability-contract`, `check:protocol-coverage`,
`check:capability-inventory`, `check:protocol-manifest` and the replay
golden check pass.
- Rust tests that read the generated contract: see the CI `runner`
checks.
- Paid narrow run: see the comment below.

## Risks

- Previous-release story: tool schemas are not saved run identity. The
OpenCode bridge lists tools again when a session resumes. A durable Rust
run that is already attached keeps the tool catalog it saved, because
re-attaching the same run does not replace the catalog. Every input the
old strict schemas accepted is still accepted, and every input they
rejected is still rejected, so saved calls, receipts and replays carry
forward. No session is reset, and no profile version or provider
`commandDigest` changes. The pack `contractDigest` changes because
`generated/` changed, as it does for any tool contract change.
- A rejected variant now arrives later for the semantic tools: the
server returns `422` instead of the runner bridge rejecting it. The
error text names the problem.
- For `paperclip_finish` on Codex, the advertised schema no longer has
the root conditional. The Rust bridge accepts a `done` result with a
continuation object, and canonical result validation still rejects it,
as before.
- Latency: the advertised schemas of the 8 tools are 1,876 bytes smaller
(23,785 to 21,909), about 470 fewer prompt tokens when all are listed.
Compiling their validators in the bridge takes 164 ms instead of 214 ms
(median of 15). The guard adds one SHA-256 of the canonical arguments
for each rejected call only. Nothing is added before turn start.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:08:58 -06:00
DottaandClaude Opus 5.5 0ece025749 perf(ci): cut repeated CPU work in the Runner support suite (#21390)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the "Runner support suite" CI job (`pnpm
test:e2e:runner:unit`)
> - That job took 8.1–10.6 minutes, depending on the spot instance type,
and was one of the last PR jobs to finish
> - The target for every PR job is 7 minutes or less
> - Measurement showed the suite is CPU-bound: each of the 693 Vitest
files loads `@playwright/test`, the runner catalog and shared modules
again in its own worker, and a few modules did quadratic or repeated
work when they loaded
> - This pull request removes that repeated work and reuses compiled
code across workers
> - The benefit is a job 75–86 s (13–16%) shorter on each instance type,
with the same files, tests and assertions. It does not reach 7 minutes;
the floor is explained below

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The duration of the "Runner support suite" PR job.

**Subsystem affected**
CI, the Runner E2E support suite under `tests/runner-e2e/`, and one
shared module (`packages/shared/src/connection-routing.ts`).

**Current behavior**
In 26 recent green PR runs the job took a median of 526 s on
`m7i-flex.xlarge` (17 runs), 549 s on `m7i.xlarge` (3 runs) and 604 s on
`m6a.xlarge` (6 runs). Vitest ran 693 files on 3 workers. The CPU was
about 82% busy. 52% of the tracked time was module evaluation.

**Proposed behavior**
The job is shorter on every instance type. The same files and tests
pass, and no assertion changes.

**Reason and benefit**
The PR workflow finishes sooner, so every pull request gets its result
sooner.

**Breaking changes**
None.

## What Changed

Measurements below are from the job's own runs on this PR. The spot
fleet picks `m7i-flex`, `m7i`, `m6a` or `m5a` (all 4 vCPU); speed
differs by up to 2x between them, so each comparison uses the same
instance type. This suite has no Vitest retries, so a flaky test fails
the job.

### Baseline (recent green PR runs on master, before this PR)

| Run (job id) | Job | Test step | Vitest wall | Vitest split |
| --- | --- | --- | --- | --- |
| 131933126599 | 523 s | 474 s | 469.8 s | import 52%, tests 43%,
transform 5% |
| 131932775003 | 510 s | 461 s | 457.1 s | import 51%, tests 44%,
transform 5% |
| 131926764360 | 527 s | 479 s | 475.2 s | import 51%, tests 43%,
transform 5% |
| 131924082813 | 541 s | 496 s | – | – |
| 131912527065 | 517 s | 468 s | – | – |

- One Vitest invocation (693 files, 3 workers, the Vitest default of
CPUs − 1), then 386 `node:test` tests in about 3 s. About 45–50 s of the
job is checkout and dependency restore before the test step.
- Summed tracked time was about 1,297 s for 470 s of wall time:
effective concurrency 2.76 of 3 workers. Module evaluation was 674 s:
4,164 modules evaluated 77,699 times, because each file runs in a fresh
isolated worker.
- A CPU sampler (temporary, removed) showed 82% average CPU use with 3
workers and 95–99% with 4 workers. With 4 workers and no other change
the wall time did not improve (455 s). The suite was CPU-bound, not
idle.
- Top modules by summed evaluation time (diagnostic run):
`@playwright/test` 158 s over 390 files, `tests/runner-e2e/catalog.ts`
56 s over 141 files, `packages/shared/src/connection-routing.ts` 40 s
over 317 files, `acpx/dist/runtime.js` 39 s.
- Slowest files: `chat-rendered-links` 43 s, `results-root` 43 s,
`chat-feature-flow` 43 s, `report` 40 s, `manual-private-transfer` 26 s,
`private-framework-report` 24 s, `chat-catalog-fingerprint` 24 s,
`hire-delegate-flow` 23 s, `source-sdk-evidence` 19 s, `unit-launcher`
16 s. No single file is near the 7-minute target, so there is no
longest-file floor.
- Slowest tests: the 256 MiB archive-cap test 23 s, the reload-once
reply wait 20 s (plus two at 10 s), the documented pnpm validation 16 s,
the retained name-only identity replay 14 s, the selected-case
fingerprint replay 13 s.

### Changes, one at a time, with what each one bought

1. **Compile cache.** `run-unit.mjs` sets `NODE_COMPILE_CACHE` to
`<tmpdir>/paperclip-runner-e2e-compile-cache` unless it is already set.
Vitest workers and the Node processes that tests start reuse V8 compiled
code. Node validates each entry against the source. `@playwright/test`
evaluation went from 158 s to 107–124 s, `acpx` from 39 s to 26–27 s,
and `zod` from 17 s to 9–10 s. Vitest went from 478 s to 441 s on a
comparable run (this run also included change 2).
2. **Reuse import scans in the chat catalog fingerprint.**
`fingerprintChatDependencyClosure` ran one esbuild parse for each of
about 2,300 files on every call. The parse result is now cached by file
path and content SHA-256 for the process. The fingerprint inputs and
digest are unchanged. `chat-feature-flow` went from 43 s to 12–14 s,
`chat-catalog-fingerprint` from 24 s to 11–13 s, and
`chat-campaign-identity` from 13 s to 8 s.
3. **Hash each suite definition once.** `buildRunnerMatrix` computed
`suiteDefinitionHash(suite)` for every execution, thousands of times,
every time a test file loaded the catalog. It now computes it once per
suite. The values are the same. `catalog.ts` evaluation went from 51 s
to 9.6 s. Vitest went from 441 s to 389 s.
4. **Fake timers for the reload-once reply waits.** These three tests in
`chat-rendered-links.test.ts` waited out the real 10 s render floor
twice. They now advance fake time with `vi.runAllTimersAsync()`. The
assertions on results, reload counts and attempt phases are unchanged.
The file went from 43 s to 2.8 s. This freed a worker slot; on its own
it did not shorten the CPU-bound run.
5. **Derive each aggregator identity once.** `connection-routing.ts`
called `aggregatorAppIdentity` (two regex replaces) for every index
entry for each of the 133 Arcade apps while it loaded, in 317 test files
and in the server. The identity of each entry is now computed once. I
compared the resulting `AGGREGATOR_SUPPORT_INDEX` (1,579 entries) and
`searchAggregatorServices` results with master: they are identical. The
SDK public input pins are refreshed for this change.
6. **Not shipped: four workers.** With the CPU work above removed, four
workers cut Vitest from 379 s to 341 s (job 418 s). In the next run,
`chat-installed-lane.test.ts` failed: the descendant of a timed-out
preparation child was still visible to `process.kill(pid, 0)` right
after cleanup. This is the process-tree-under-load risk, so the suite
keeps the Vitest default of 3 workers.

Docs: `tests/runner-e2e/README.md` describes the compile cache, how to
turn it off, why the worker count stays at the default, and the cost of
work at module load.

### Result

| Run | Job | Test step | Vitest wall | Effective concurrency |
| --- | --- | --- | --- | --- |
| Baseline median, `m7i-flex` (17 runs) | 526 s | 465 s | 462 s | 2.76 |
| Baseline median, `m7i` (3 runs) | 549 s | 476 s | 473 s | – |
| Baseline median, `m6a` (6 runs) | 604 s | 535 s | 531 s | – |
| This PR, `m7i-flex` (job 132007703795) | 440 s | 383 s | 379 s | 2.79
|
| This PR, `m7i` (job 132027384596) | 474 s | 424 s | 420 s | 2.78 |
| This PR, `m6a` (job 132033745288) | 526 s | 471 s | 466 s | 2.79 |
| This PR, `m5a` (job 132040621237; dependency cache miss) | 880 s | 768
s | 761 s | – |

The last three rows are the final configuration. The `m7i-flex` row is
the same code before a README-only change. Module evaluation fell from
77,699 to 63,135 evaluations; part of that drop came from master changes
merged into this branch, so I attribute time only to the per-change
measurements above.

## Verification

- CI on this PR is the experiment. Every push ran the real job on the
real runner. The table above lists each run.
- Local correctness checks, one process at a time:
`chat-rendered-links.test.ts` (58 passed), the six chat fingerprint test
files (37 passed), `catalog.test.ts` (92 passed), `packages/shared`
`connection-routing.test.ts` and `connection-search.test.ts` (21
passed), and `node tests/runner-e2e/run-unit.mjs
connection-auth.test.ts` (launcher path, 386 node:test passed).
- Equivalence check for the connection routing change: the index and
search results are byte-identical to master.
- Flakiness: 10 runs of this branch with 3 workers (including the three
final runs on `m7i`, `m6a` and the 2x slower `m5a`) passed with 0 failed
tests and 0 timeouts. No port conflicts, OOM kills or `ps` timeouts
appear in the logs. The suite has no retries. The 4-worker setting
failed once in 3 runs, so it is not used.

## Risks

- Low risk. No test is skipped or removed. No assertion is weakened.
- The worker count is unchanged, so the load on process, port and
temp-directory tests is lower than before (less CPU work per file).
- The compile cache directory lives in the OS temp directory. Node
rejects a stale or partial entry and compiles again.
`NODE_DISABLE_COMPILE_CACHE=1` turns it off.
- The connection routing change runs in the server too. It produces the
same index, and it makes loading the module cheaper.

### What sets the remaining floor

The job is about 45–55 s of checkout and dependency restore plus the
Vitest run. The Vitest run is CPU-bound on 4 vCPU (2 physical cores). A
fourth worker gives about 40 s more but fails a process-tree cleanup
test under load (`chat-installed-lane.test.ts`, which checks the
descendant immediately after cleanup). The largest remaining cost is
per-file isolation: every file loads `@playwright/test` (about 0.3 s CPU
each, 390 files), `zod` schemas from `packages/shared`, and `acpx`
again. Vitest estimates that `isolate: false` would save about 140 s
now. I did not use it: 116 files use `vi.mock`, and module state shared
between files would make failures depend on which files share a worker.
If more headroom is needed, the next step is a split across two machines
(out of scope here because it changes `.github/`). Two duration-balanced
halves (`vitest --shard=1/2` and `--shard=2/2`, about 190–235 s of
Vitest each on `m7i-flex`/`m6a`) would put each job at about 4–5 minutes
on every instance type except `m5a`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable (one test moved to
fake timers; behavior changes are equivalence-checked)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:05:05 -06:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
DottaandClaude Opus 5.5 2de4787f41 fix(db): release a postgres.js connection's receive buffer once it is fully read (#21432)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reads PostgreSQL through postgres.js, with a patched
driver in `patches/postgres@3.4.9.patch`
> - A hosted server keeps about 0.5 GB of `external` memory (Buffers)
for its whole life, with 340 open database connections
> - postgres.js keeps a zero-length view of the buffer of each
connection's last read. The view keeps the whole buffer alive until that
connection reads again
> - An idle connection therefore pins the socket chunk or the whole
buffer of its largest recent message, which can be one MB or more
> - This pull request replaces a fully read buffer with an empty one in
the driver patch
> - The benefit is that idle connections hold no receive buffers. At 340
connections that each read a 1 MB row, external memory drops from about
320 MB to 0

## Linked Issues or Issue Description

**What happened?**
A hosted Paperclip server holds 455 to 520 MB of `external` memory for
its whole life. It has about 340 open database connections. Each idle
postgres.js connection keeps the buffer of its last read.

**Steps to reproduce**
1. Open 80 or 340 pool connections.
2. Read a 1 MB row on each connection.
3. Drop the results, let the connections go idle, and force garbage
collection.
4. Measure `process.memoryUsage().external`. Each idle connection still
holds about 0.95 MB.

**Expected behavior**
An idle connection holds no buffer from a result that it has completely
read.

**Environment**
postgres.js 3.4.9 with the Paperclip patch, Node.js 24 and 26,
PostgreSQL 17/18.

## What Changed

- `patches/postgres@3.4.9.patch`: in `data()` of `src/connection.js` and
`cjs/src/connection.js`, the driver now replaces the receive buffer with
an empty buffer when it has handled every message in it. A partly read
buffer is not changed. The other hunks of the patch are unchanged. I
regenerated the patch with `pnpm patch postgres@3.4.9`, edited the two
files, and ran `pnpm patch-commit`. I did not commit the lockfile change
from that command.
- `packages/db/src/postgres-receive-buffer.test.ts` and
`packages/db/src/__fixtures__/postgres-receive-buffer.mjs`: a regression
test for ESM and CommonJS on real PostgreSQL.
- It checks that results are reassembled correctly for a 3 MB row, for
161 row sizes around one 64 KB socket chunk (to land message boundaries
at the end of a chunk), and for 20,000 small rows that arrive many to a
chunk.
- It records a weak reference to each large receive buffer and checks
that none of them stays alive after the connection becomes idle.
- `packages/db/scripts/benchmark-postgres-receive-buffer.ts`: one
command that reports heap, external and rss per connection, before and
after large rows.
- `doc/DATABASE.md`: documents the new patch behavior, the test and the
benchmark.

## Verification

- Red/green: with the new line removed from the installed driver, the
fixture fails for ESM and CommonJS ("an idle connection must not keep
the receive buffer of its last result"). With the line in place it
passes, repeated twice for each format.
- `pnpm --filter @paperclipai/db exec vitest run
src/postgres-receive-buffer.test.ts
src/postgres-connection-recovery.test.ts`: 20 passed.
- `pnpm --filter @paperclipai/db exec vitest run src/backup-lib.test.ts
src/backup-lib-binary.test.ts src/client-disconnect-replay.test.ts
src/client-teardown-registry.test.ts src/work-signals.test.ts`: 36
passed. In 1 of 4 runs, the JavaScript backup test (COPY and cursor
reads) timed out on a machine with load average 20 to 34. It took 39 s
and 25 s with the change and 21 s without it in passing runs. Read
throughput is the same with and without the change: a 200,000-row select
plus a COPY TO STDOUT has a median of 319 and 348 ms with it, and 329 ms
without it.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/prepared-statements-driver.test.ts
src/__tests__/prepared-statements.test.ts`: 5 passed.
- `pnpm --filter @paperclipai/db typecheck`: passed.
- Benchmark: `node --expose-gc --import tsx
packages/db/scripts/benchmark-postgres-receive-buffer.ts` with a 1 MB
row on each connection, numbers after garbage collection:

| Connections | External per idle connection, before | After | External
held in total, before | After |
| --- | --- | --- | --- | --- |
| 80 | 917 KB | 1 KB | 72 MB | 0 MB |
| 340 | 960 KB | 0 KB | 319 MB | 0 MB |

The rss per connection also went down (80 connections: 1,555 to 1,375
KB; 340 connections: 1,308 to 714 KB). The allocator keeps some freed
memory, so rss changes less than external memory.

## Risks

- Low risk. The change runs only when the buffer has no unread bytes. A
message split across chunks keeps the same reassembly path. The test
covers split messages, exact chunk boundaries and many messages in one
chunk.
- This saves external memory (Buffers outside the JavaScript heap). It
does not reduce `heapUsed`. It does not fix the out-of-memory crashes of
the hosted server by itself.
- The patch must be removed or ported when Paperclip upgrades
postgres.js. The test fails if a driver version keeps the buffer again.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), used in
Claude Code with tool use (shell, file editing, local PostgreSQL), heap
snapshots and allocation measurement.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:38:32 -06:00
DottaandClaude Opus 5.5 594f9d3ec2 perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee

## Linked Issues or Issue Description

**What happened?**

A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.

**Expected behavior**

Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.

**Steps to reproduce**

1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.

`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.

**Paperclip version or commit**

`master` at 2659ff1d22.

## What Changed

- `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits
for the write in progress. A newer flush for the same run replaces that
snapshot and returns the same promise. A flush for a different run (a
warm handoff) still queues its own write.
- New tests in `durable-prp-control-plane.test.ts`:
- 40 commands queued behind a slow write produce one more write, not 40.
Each caller resolves only after the stored records contain its command.
The stored records decode to the last written checkpoint.
- When the combined write fails, every caller that waited on it rejects,
and the next flush writes a full reset that contains the commands.
- New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained
heap per run, checkpoint writes, and whether finished control planes are
collected.

Why this is safe:

- A waiting snapshot has not started its write. No caller has received a
result for it, so no acknowledgement or command send depends on it.
- The replacement is captured from a later state of the same store. It
contains every change that the replaced snapshot contained.
- Every caller of the replaced snapshot waits on the same promise. The
promise resolves only after the newer snapshot is durable. A failure
rejects all of them, and the existing retry writes a full reset.
- Writes stay on one chain, in order. The server fence, the encryption,
and the stored checkpoint format do not change.
- A flush for a different run ID never replaces a waiting snapshot, so
the earlier run still gets its last state.

## Verification

- Red/green: the two new tests fail on `master` (42 writes instead of 3;
the waiting callers resolve instead of rejecting) and pass with the fix.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
in `packages/paperclip-runner`: 219 of 219 pass.
- `node scripts/check-module-boundaries.mjs` passes.
- Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s
writes, 300 events of 2 KB already in each run:

| | `master` | this PR |
| --- | --- | --- |
| Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB
|
| Retained heap per idle live run | 1,296 KB | 1,415 KB |
| Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) |
| New events committed per run in 30 s | 4 | 30 |
| Control planes collected after stop | 8 of 8 | 8 of 8 |

- With commands at 0 per second, `master` has no backlog: an inbound
event waits for its own checkpoint before the next frame is read. The
backlog comes from flushes that do not wait.

## Risks

- Behavioral shift: when a combined write fails, callers whose own
earlier revision would have been written alone now see that failure. A
checkpoint failure already disconnects the runner and the next flush
writes a full reset, so recovery is the same.
- Fewer intermediate states reach the database. A crash can only lose a
state that no caller has seen as saved, which is the same as before.
- Not covered by a test: a warm handoff whose run ID changes while a
snapshot waits. The code does not combine those; the existing
warm-transition tests pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:08:15 -06:00
DottaandClaude Opus 5.5 e7e28ba241 feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns`
tasks at once. Other runs wait in the agent's queue with status `queued`
> - On hosted staging, one agent had all 20 slots full and 295 queued
runs. Some tasks waited 3 to 7 hours and never started
> - The UI showed these tasks as live or working: a pulsing "Live" badge
in the task header and in task rows, a spinning "Working … for 4 hours"
in the task chat, and "N live" counts that included every queued run
> - People could not tell that work was waiting, why it waited, or what
to change
> - This pull request makes a deep queue an honest, visible state:
"Waiting for Codie · 20 slots max · 41 of 295 in line · Configure"
> - The benefit is that operators see the real state and the one setting
that changes it

## Linked Issues or Issue Description

Refs #21391, #21349, #21242

**What happened?**

A queued run that never started looked the same as a running run. The
task header, the task rows in lists and the inbox, the task chat and the
sidebar counts all showed queued runs as live or working.

**Expected behavior**

A run that waits for one of its agent's concurrency slots shows that it
waits. It names the agent, the agent's limit and the run's place in
line, and it links to where the limit is set. The place in line is the
order in which the server starts the agent's queued runs.

**Why does it matter?**

With a deep queue, the UI said "working" for hours while nothing ran.
Operators could not see the cause or the fix.

## What Changed

- Server: a new module
`server/src/services/heartbeat/queued-run-line.ts` now holds the facts
that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The
dispatch path in `queue.ts` and the new "place in line" read use this
one builder, so the line the UI shows is the dispatch order. The logic
moved without change, including the recovery-share fix from #21391.
- Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit,
running count, position, queued count) and the label "Waiting for
<agent>" with cause `agent_at_capacity` to a queued, never-started run
when its agent has no free slot. It reads one batch of agents and
running counts. Then it builds one line per full agent (at most 8 agents
per read), using only the three context keys that ordering needs. No new
polling.
- Server: the place in line uses `queuedRunDispatchLine`, an O(n log n)
replay of the dispatcher. The plain replay rescans the queue after each
step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000,
600 ms at 10,000, all on the event loop). Dispatch itself is unchanged:
it stops after the free slots. Each agent's line is memoized for 5 s and
shared by concurrent reads. In a preview worktree, the line applies the
same creation-time cutoff as dispatch.
- Server: the task list summary adds `runningDescendantCount`
(descendants whose run started) next to `liveDescendantCount`.
`liveDescendantCount` still decides whether a blocked parent is covered.
The "n live below" chip uses the new count.
- Shared: the new `ExecutionAgentQueue` type and the optional field
`ExecutionProjection.agentQueue`.
- Server: `/companies/:companyId/live-runs` lists running runs before
queued runs, so a deep queue cannot push running work out of the 50-row
limit.
- UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20
slots max · 41 of 295 in line · Configure", with no spinner and no
elapsed timer. Configure opens the agent's Advanced view, where the run
policy (`maxConcurrentRuns`) is edited. The task chat (both renderers),
the run ledger and the Storybook story use it.
- UI: the task header and the task rows (task list, inbox, legacy views)
show a quiet "Waiting" instead of the pulsing "Live" when every live run
of the task has not started. The full sentence shows on hover. "n live
below" counts only started sub-tasks. A waiting sub-task still covers
its blocked parent.
- UI: the sidebar "N live" counts (Dashboard, agents) count only started
runs.
- UI: the task chat pill now names other server holds on a queued run
(for example "Waiting for maintenance") instead of "Working".
- Docs: DESIGN.md records the waiting-for-slot hold.
- Storybook: `runs-agent-queue-wait` stories (full line, task chat, task
row, one slot, mobile), with play checks.

## Verification

- Red/green server test:
`server/src/services/execution-projection-agent-queue.test.ts` (embedded
Postgres). Before this change, the projection has no `agentQueue`. Now
it checks the agent name, limit, running count, positions 1 to 4 in
dispatch order (a person's retry first, then priority, not creation
order), no wait while a slot is free, and isolation from other
companies' runs. It also checks that the positions equal the line that
the dispatch builder produces.
- `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`,
`recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and
`agent-live-run-routes` in `server/`.
- Runtime cost (rule `runtime-dependency-boundaries`): the assumption is
that the line stays cheap for a deep queue. `queued-run-line.test.ts`
checks that `queuedRunDispatchLine` equals the dispatcher's
claim-every-run order over 400 random queues (ages around the aging
boundary, recovery and person runs, blocked runs, deferred share runs,
all share counters). It also orders 10,000 runs in under 250 ms (the
quadratic replay takes about 600 ms). The DB test projects a 300-run
queue end to end with correct positions 1 to 300. Remaining limit: the
line is up to 5 s stale.
- Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts
`runningDescendantCount: 0` for a parent whose only live descendant is
queued and never started.
- UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep
"Live", no "running below" count for waiting sub-tasks) and
`TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the
Configure link, other holds named). The suites for the run ledger,
sidebar, sidebar agents, task detail, task columns and task chat also
pass.
- The Storybook stories render in headless Chromium and their play
checks pass.
- After deploy: open a queued task on staging and confirm the waiting
state.

## Risks

- Cost on read: a projection read that includes a queued run of a full
agent now reads that agent's queued runs (only small columns) and the
dispatch facts. This is at most one line per full agent every 5 s, for
at most 8 agents per read. Reads with no queued, never-started runs do
no more work.
- The place in line is the server's prediction of dispatch order. A
claim that is refused at start time (a gate, a blocker) can let the next
run start first. The position updates on the next refresh.
- The process-local "declined aged claim" hint that the dispatcher uses
is not visible to other servers, so the projection does not use it. This
can change only the aged-oldest share step.
- Configure opens the agent's Advanced view (the YAML run policy). The
agent's Runtime tab hides the run-policy fields today.
- Low risk otherwise: no schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless browser) and subagents for the UI survey.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:24:20 -06:00
DottaandClaude Opus 5.5 b7774a01cf fix: name activity actors and explain stopped tasks in plain words (#21406)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People must never see internal jargon or raw ids in chat replies or
task comments
> - Chat eval cohort 12 found three sources: activity records show the
board user only as the raw id `local-board`, agents turn internal field
names such as `checkoutRunId` into words like "no checkout", and a
recovery notice says "the original assignee is not invokable"
> - The recovery notice appeared on 170 fixture tasks. It is not a loop:
each task got it once, because recovery blocked every in-progress task
of an agent that a person had paused
> - This pull request names activity actors, lists these internal terms
in the chat guidance, rewords the notice, and leaves in-progress work of
a manually paused agent alone
> - The benefit is plain replies and no false "blocked" tasks after a
person pauses an agent

## Linked Issues or Issue Description

**What happened?**
- `activity.audit-change`: the agent read `GET
/api/issues/{id}/activity` and wrote that the change was made by "the
board user identity (`local-board`)". Activity rows carry only
`actorType` and `actorId`.
- `activity.company-overview`: the agent wrote "no checkout" to explain
why no one is working on a task.
- `tasks.organize-open-work`: 170 of 501 fixture tasks got one system
comment each: "Paperclip cannot safely continue automatic recovery
because the original assignee is not invokable...". The owners were
paused by a person, and the stranded-work sweep blocked each in-progress
task. The existing rule already keeps `todo` work of a manually paused
agent, because blocking it fights the pause and strands the task after
resume.

**Expected behavior**
Replies name actors by name ("Board", a member's name, an agent's name)
and describe states in plain words. A person's pause does not block the
agent's in-progress work. A stopped-task notice says what happened and
what to do, in plain words.

**Steps to reproduce**
1. Change a task's priority as the local board user, then ask the chat
agent who changed it.
2. Pause an agent that owns in-progress tasks with no active run, and
let the stranded-work sweep run.

## What Changed

- `server/src/services/activity.ts`, `server/src/routes/activity.ts`,
`packages/shared/src/types/activity.ts`: company and task activity reads
add `actorName` (agent name, member name, "Board" for `local-board`,
"Paperclip" for the system). Raw ids stay in their existing fields.
- `server/src/services/agent-conversations.ts`: the existing "no raw IDs
or internal codes" rule now names checkout, native run and local-board
as internal terms, and tells the agent to name an actor by `actorName`.
- `server/src/services/recovery/service.ts`: the notice is now "<agent>
can't run right now, so Paperclip stopped this task instead of retrying
it. It is still assigned to them. To continue, resume or fix the agent,
or assign the task to someone else." The manual-pause skip now covers
`in_progress` as well as `todo` work with no run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 4 pass. With
the old recovery service the two new tests fail (`expected 'blocked' to
be 'in_progress'`, and the old notice text).
- `cd server && npx vitest run src/__tests__/activity-service.test.ts -t
"names each activity actor"`: passes (the method did not exist before).
- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: passes.
- `src/__tests__/activity-routes.test.ts` has three tests that time out
locally on unchanged master code too; CI runs them.
- Paid eval run of `activity.audit-change` and
`budgets-costs.spend-breakdown` on a branch with this change.

## Risks

- The chat prompt grows by about 40 words. This is a small part of the
existing directive, so time to first token does not change in a
measurable way.
- A budget- or error-paused agent still gets the notice, now in plain
words. Only a manual pause keeps in-progress work as is; the sweep picks
it up after resume.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:53:25 -06:00
DottaandClaude Opus 5.5 944b1c64d7 feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use connected apps through Paperclip's MCP client in the tool
gateway
> - Some provider actions run for minutes: CI runs, deploys, batch jobs
> - Today such a call is lost if it outlives the tool timeout or if
Paperclip restarts or deploys mid-call, and a retry can start the work a
second time
> - Recovery must just work, and the target rate of runs that need a
manual retry is 0%
> - This pull request implements the MCP Tasks extension
(`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client
> - The benefit is that a long provider call becomes a durable task: it
survives timeouts, restarts, and deploys, and it is never started twice

## Linked Issues or Issue Description

**Problem**
A long provider tool call holds an HTTP request open. When it passes the
tool timeout, or when Paperclip restarts or deploys during the call, the
result is lost. The agent then often repeats the call, and the provider
does the work again.

**Proposed solution**
Implement the client side of the MCP Tasks extension. Declare it on
stateless tool calls. Store each returned task durably, wait on it
during the call, keep polling it in the background after the call ends,
and let the same call attach to the stored task instead of sending
`tools/call` again.

**Why it matters**
CI, deploy, and batch actions are where agents lose the most work today.
A durable task handle makes these actions safe across timeouts,
restarts, and deploys, without a person.

Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP
client, and MCP Skills work this builds on).

## What Changed

- **Off by default:** MCP Tasks are behind the new instance experimental
setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default
off. When it is off, Paperclip never declares the extension, nothing
attaches to or polls tasks, pending rows do not block idle sleep, and
behavior matches master. A test covers the off state.

- `mcp-http.ts`: the tasks extension is declared per request, only on
HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers
do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`.
- New `mcp-tasks.ts`: parses task results, bounds the server's timing
hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals
task IDs at rest (AES-256-GCM, bound to company and connection; a keyed
hash gives uniqueness).
- New company-scoped table `tool_mcp_tasks` (migration `0475`,
online-compatible). It links the task to the call, run, task,
connection, and grant. It stores no provider result.
- `tool-gateway.ts`:
- A returned task is recorded, then polled up to the normal tool
timeout. Its result goes through the same validation, redaction, and
recording as a direct result.
- Past the timeout, or when the caller disconnects, the agent gets
`provider_task_pending` ("call again with the same arguments"). The task
stays durable.
- A call with the same endpoint, tool, full arguments, grant, agent,
task, and actor attaches to the undelivered task. A retry with the same
idempotency key also attaches, and it can never start different work
under that key.
- Input: URL requests show the provider link (Paperclip never answers
them). Form requests become a question card; only the person's answer is
sent with `tasks/update`. Other requests stop with a clear "needs input"
error.
- Stopping the run sends `tasks/cancel` and marks the task cancelled. A
caller that only disconnects keeps the task.
- `failed` becomes a tool error with the provider's message. An unknown
task or a passed TTL becomes `provider_task_expired`. Every task stops
being tracked at its TTL, or after 7 days at most.
- Audit entries: `tool_gateway.task_started`, `task_input_sent`,
`task_cancelled`.
- New `mcp-task-work.ts`, registered at startup with the shared deadline
scheduler: it resumes polling pending tasks after a restart, cancels
tasks whose task closed or run was stopped, and wakes the assignee once
when a task settles after its run ended.
- Docs: `doc/connections/MCP-TASKS.md`, linked from
`doc/connections/README.md`.

Background work (per `rules/sleep-aware-background-work.mdc`): trigger
is a task row write or release (`mcp-tasks` queue) and one startup pass.
The deadline is the earliest `next_poll_at`. Cadence is the server's
`pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting
for a person. It is quiet when no row has a next poll. This is the
narrow external-source exception: providers give no wake Paperclip can
receive, and polling stops with the task.

## Verification

- `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake
2026-07-28 server that returns tasks only to a client that declares the
extension and checks `Mcp-Name`. They cover:
- polling to completion, with the result identical to a synchronous
call;
  - timeout, then an attaching call with no second `tools/call`;
  - a keyed retry, and a reused key with different arguments (refused);
- restart mid-task: a fresh gateway with no in-memory state polls the
row, wakes the agent once, and the recovered run gets the result once
with no second `tools/call`;
  - no wake while the run is still going;
- URL input (link, never answered), and form input (question card, then
`tasks/update` with the answer);
- cancellation by the run, by closing the task, and no cancellation when
only the caller leaves;
- failure, an unknown task, and TTL expiry (ends in under 3 s, no hang);
  - another agent never attaches;
- synchronous calls unchanged and older servers never see the extension.
- Red/green: with attach disabled, the timeout, keyed-retry, and restart
tests fail (3). With `tasks/cancel` disabled, both cancellation tests
fail (2). Before `tools/call` declared the extension, 13 end-to-end
tests failed. All pass with the change.
- Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`,
`mcp-skills`, `server-startup-feedback-export`, `tool-gateway`,
`tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests).
- `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db
check:migrations` passes (the migration is classified
online-compatible). Snapshots pruned to five.
- Agent boot and time-to-first-token: no change in the boot or
first-turn path. The only new per-call cost is one indexed lookup on
stateless tool calls.
- Not run: a live provider that returns tasks. No public MCP server that
returns tasks was available without new credentials.

## Risks

- Shipped off by default. Review findings that only affect the
setting-on path after this merge are tracked as follow-ups.

- Migration: a new table with indexes and foreign keys to `companies`,
`tool_invocations`, and `tool_connections`. Online-compatible.
- Behaviour change only for servers that return tasks, which they may do
only to clients that declare the extension. Older servers and stdio
servers are unchanged.
- A task that settles while nobody is waiting stays attachable for 1
hour, so a deliberate repeat of the same call within that hour collects
the earlier result instead of starting new work.
- The wake after a background task settles uses the generic task wake;
the agent must call the tool again to collect the result.
- Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling
and roots input requests; showing a long action's progress in the task
chat.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:32:06 -06:00
DottaandClaude Opus 5.5 8af889f612 feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP Apps (#21119) show interactive widgets from MCP servers, such as
Circleback meeting cards, with the agent's reply in the task chat.
> - The first version deferred three parts of the MCP Apps spec. A
widget could not propose a chat message (`ui/message`). It could not
give the agent context (`ui/update-model-context`). It could not go full
screen.
> - Paperclip also ignored the `icons` that MCP servers declare on tools
(SEP-973).
> - The security review of #21119 showed what these features need from
day one: explicit consent, untrusted-data labels and strict icon
sanitizing.
> - This pull request adds the three host features and provider tool
icons, with those protections built in.
> - The benefit is that widgets can now work with the person and the
agent, not only display data.

## Linked Issues or Issue Description

Refs #21119

**Subsystem affected**
MCP Apps host (task chat widgets), tool catalog, run wake prompt.

**Problem or motivation**
The MCP Apps host refused `ui/message` (-32601), dropped
`ui/update-model-context` and kept every view inline. MCP tool `icons`
were not parsed or shown. So widgets could not ask the agent a
follow-up, the agent could not see what the person selected in a widget,
large widgets were cramped, and tool rows used only generic icons.

**Proposed solution**
Add consent-gated view messages, per-view app context delivered to the
task's next run as untrusted data, a full-screen dialog, and sanitized
provider icons. Details are under "What Changed".

**Alternatives considered**
- Send view messages with a new server route that creates the comment
itself. Rejected: comment creation has many rules (agent chat ownership,
paused conversations, queued delivery, wakeups, receipts). The widget's
Send button uses the same send path as the composer. A small route then
records which view the comment came from.
- Keep the iframe in place for full screen with CSS. Rejected: chat
containers can create containing blocks that break `position: fixed`. A
moved iframe always reloads, so the dialog hosts a new frame instead.

**Roadmap alignment**
Continues the MCP feature work after #21119, #21140 and #21219. It does
not touch the MCP Tasks work (gateway `tools/call` polling and
elicitation).

## What Changed

- **`ui/message`.** The host shows "<App> wants to send: …" with
**Send** and **Dismiss** under the widget (and inside the full-screen
dialog). Nothing is sent before the click. Send posts the text as the
viewer's own comment through the chat's normal send path
(`handleThreadAdd`). `POST
/api/issues/:issueId/mcp-app-views/:viewId/messages` then logs
`issue.mcp_app_message_sent`. It accepts only the caller's own comment
on the same task, matched by `clientRequestId`, from the last 10
minutes. One proposal per view can wait at a time. The view gets `{}`
after the send, or -32000 "Message sending denied" after Dismiss. Only
`user` text up to 8,000 characters is accepted.
- **`ui/update-model-context`.** `PUT
/api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the
latest context per view in `tool_mcp_app_views.model_context` (text
blocks and `structuredContent`, 16 KB at most). Updates are applied in
order and deduped in the browser and on the server. A change logs
`issue.mcp_app_context_updated`, and an empty update clears the context.
The task's next run gets the three most recently updated contexts as an
`## App context (untrusted)` wake-prompt section. The section quotes the
data as JSON and says it is third-party data, not instructions. The read
runs in parallel with the other run preparation and never fails the run.
CLI adapters and the native runner render it through the same
`renderPaperclipWakePrompt`.
- **Who may do this.** Messages and context need a signed-in person with
`issue:comment` on the task. Agents and API keys get 403. Other tasks
and companies get 404. `message` and `updateModelContext` are advertised
in `hostCapabilities` only where the viewer can send messages.
- **Full screen.** The host advertises `availableDisplayModes:
["inline","fullscreen"]` and handles `ui/request-display-mode`. It
honors the view's declared modes and answers with the resulting mode. It
sends `host-context-changed` with `displayMode` and the container (fixed
`height`/`width` in full screen), and again when that container resizes.
A new **Open full screen** control sits next to "Widget from …". Full
screen is a Paperclip dialog that closes with Exit, Escape or the view's
own request.
- **Provider tool icons.** `icons` on MCP tool descriptors are sanitized
into the new `tool_catalog_entries.icons` column, outside the version
hash, so a new icon never quarantines a reviewed tool. Only `https:`
URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A
`data:` icon is at most 32 KB, a URL at most 2,048 characters, with at
most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists
them keyed by gateway tool name, in one cached request per chat. Task
chat tool rows and the app's actions list show them through
`ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a
fixed size and never inlines markup. It checks the source again before
rendering and falls back to the taxonomy icon.
- **Schema.** Migration `0477` adds `tool_catalog_entries.icons`,
`tool_mcp_app_views.model_context` and `model_context_updated_at`, all
nullable.
- **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new
behavior, and the finished items are removed from "Limits and
follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a
message", "Full screen" and "Tool rows with provider icons". The fixture
view now uses `ui/message`, `ui/request-display-mode` and
`ui/update-model-context`.

## Verification

- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/shared/src/mcp-icons.test.ts
packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These
tests cover:
- Context: stored, deduped, logged once, 413 when too large, cleared,
scoped to the task and company, refused for agents (403), and rendered
into the wake prompt as untrusted data.
- Messages: attributed only to the viewer's own recent comment on the
same task. A teammate's comment, an unknown request id or an old comment
gets 404. An agent gets 403.
- Icons: stored sanitized, and the version hash and status do not
change. Another company sees nothing.
- `cd ui && pnpm exec vitest run src/components/mcp-app/
src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69
pass. These tests cover:
- The host protocol for each new method: capabilities, consent wait, one
proposal at a time, invalid input, ordered and deduped context, the size
limit, and the display-mode rules.
  - The consent card (nothing sent without Send).
  - The full-screen dialog (one frame at a time).
  - Icon rendering (no-referrer `img`, no inline SVG, fallback).
- `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for
the server: clean.
- Red/green: the new route and host tests fail on `master`, where the
routes do not exist and the host answers -32601 or drops updates. The
icon descriptor test failed when the publication write was removed.
- Not run locally: the full suite, and
`packages/adapter-utils/src/server-utils.test.ts › runChildProcess ›
does not crash when the child closes stdin before a delayed write`. That
test times out on this Mac with or without this change, so it is a local
platform issue. CI covers both.

## Risks

- Migration risk is low: three nullable columns, no backfill.
- Every run on a task now does one extra indexed read, in parallel with
the other preparation. Errors are logged and the run continues without
app context.
- If the attribution request fails after the comment is sent, the
comment stays without the `mcp_app_message_sent` activity row. This is
documented.
- Switching between inline and full screen restarts the view, so state
inside the widget resets. This is documented as a limit.
- Icon names shortened with a hash by the gateway (very long names) do
not match, and those rows keep the taxonomy icon.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. One subagent (the same model) built the provider-icon
part.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:32:48 -06:00
DottaandClaude Opus 5.5 d9b42b50fd feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and continues the task by itself. Since #21353 the task reads as
continuing (`working`, no Retry) for as long as that continuation waits
> - That is truthful, but a person has no control while it waits. On
hosted staging one agent had 21 running and 209 queued runs, and
continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a
waiting continuation into the agent's queue within minutes, so most of
that wait is in the queue, behind other work
> - #21353 removed Retry from this state because a Retry started its own
run, which raced the continuation and could give the task two runs
> - A Retry that observes the queued continuation run cannot start a
run: admission sees a queued run and joins the Retry to it (`coalesced`,
`runId` = that run). Under #21234's ranking, a queued run that a person
joined starts ahead of system recovery and ordinary work
> - This pull request keeps the continuing state and, once the queued
continuation has waited 15 minutes, also offers Retry labelled "Move it
forward now"
> - The benefit is that a person can move a long-waiting task to the
front of its agent's queue, and the click can never create a second run

## Linked Issues or Issue Description

Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200

**What happened?**

On hosted staging, tasks with a recorded continuation waited 30 to 110
minutes behind a deep agent queue. The task showed "continuing" with no
action. The person could only wait.

**Expected behavior**

The task still never shows "failed". After a long wait the person can
move the waiting continuation forward. The click must not start a second
run.

## Which states a Retry joins (established before building)

I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its
tests, and drove the real click path (`requestTaskRetry` followed by
`resumeTaskRetry`, as the route does) in a probe against each record
state:

| Continuation state | What a Retry does today | Offer Retry? |
|---|---|---|
| Recorded, waiting for delivery (`continuationDelivery: "pending"`, no
run yet) | The Retry is admitted at once as its own `task_retry` run,
for every isolation (`fresh_isolated_workspace`, `writer_ended`,
native). The delivery pass then marks the record delivered (superseded).
#21348's fold never runs: it folds only a Retry that is still
`deferred_issue_execution` when the continuation run is created, and
only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred,
so the click races the delivery pass. | **No** (unchanged) |
| Delivered, continuation run `queued` | `requestTaskRetry` sees the
latest run is busy and records the Retry as `coalesced` into that run in
the same transaction. No run is created; `resumeTaskRetry` acts only on
deferred Retries. The run is now ranked as a person's (`joinedByUser` in
`queue.ts`). | **Yes, after 15 minutes** |
| Continuation run `running` | Same join, but the run already started,
so there is nothing to move forward. | No |
| Continuation ended | A Retry gets its own run (the existing failed-run
path). | Existing behaviour |

So the Retry offered here is joined by the existing busy-run admission
in `requestTaskRetry`, not by #21348's fold. Its source run is the
queued continuation itself (the run the presentation names as
`observedRunId`), not the run the continuation was created for.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: for a task
whose latest run is `queued`, created 15 minutes ago or more, and named
as `continuationRunId` by a delivered fresh-continuation record, the
projection returns `state: "working"`, `canRetry: true`, `retryAction:
"move_forward"`, responsibility `system`, `observedRunId` = that queued
run. It does not offer this when a person's wake already joined the run,
when the run's agent is not the assignee, or when a person-owned wait
applies (approval, reconnect, budget, pause and the others are checked
first). Two small reads run only for queued runs older than 15 minutes.
The existing restore-hold check that withdraws `canRetry` also withdraws
`retryAction`.
- `packages/shared/src/types/execution-projection.ts`: new optional
`retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It
only changes the label; `canRetry` still decides whether Retry is
offered.
- `ui/src/components/ExecutionBlockerNotice.tsx`: a `working`
presentation with `retryAction: "move_forward"` shows "Waiting for the
agent to finish other work. Paperclip continues this task by itself."
and a "Move it forward now" button that calls the same task Retry.
Token-only classes; `pnpm check:token-gates` is clean.
- One join-only action, `move_forward`, carried from the offer to
admission (contracts kept in sync):
- `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts
`retryAction: "move_forward"`. The route already passes the body to
`requestTaskRetry`.
- `server/src/services/task-retry.ts` (`admitTaskRetry`): right after
the stale check, still under the task row lock and before any other
Retry path, a `move_forward` Retry whose observed run is no longer
`queued` or `running` returns `{ accepted: false, alreadyMovedOn: true,
requestId: null }` and writes nothing. This block was cleared with the
owner of Retry admission. The owner will later extend it so that
"continuation recorded, not yet delivered" also counts as actionable,
after their pending-delivery change lands (about two lines on their
side). Until then this presentation never offers the button in that
state, so the early return cannot misfire from our UI.
- `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click
sends `retryAction: "move_forward"` when the offer has it.
`alreadyMovedOn` shows "The task already moved on, so nothing new was
started. It now shows its current state."
- What a click does at each timing: continuation `queued` → joined to
that run (moves it forward); `running` → joined (nothing more to do);
ended → "already moved on", nothing started.
- A joined click cannot become a run later. The busy path writes it as
`coalesced` with `runId` set to the continuation and `finishedAt` set
(`task-retry.ts:456-461`). Promotion after a run ends
(`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`;
`modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and
the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`)
and `resumeExecutionWaitComments` (`:876`) pick up only
`deferred_issue_execution` wakes.
- `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry
for this state. That row sends the failed run as the observed run, which
is no longer the latest run, so its Retry would be refused as stale.
- `doc/execution-semantics.md`: one paragraph on the offer, when it
ends, and why there is none before delivery.

## Verification

- Red (before the product change, new tests only), in `server/`: `pnpm
exec vitest run src/__tests__/task-interruption-automatic-path.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both
failures are the missing offer (`canRetry: false`, no `retryAction`) for
a continuation queued 15+ minutes.
- Green after merging master 8e41247b65, in `server/`: `pnpm exec vitest
run src/__tests__/recovery-wake-herd.test.ts
src/__tests__/task-interruption-automatic-path.test.ts
src/services/task-retry.test.ts
src/services/task-interruption-presentation.test.ts
src/__tests__/task-interruption-query.test.ts`: 5 files, 279 tests pass.
UI: `useTaskRetry.test.tsx` and `ExecutionBlockerNotice.test.tsx`: 35
tests pass.
- New tests:
- (b, integration, `recovery-wake-herd.test.ts`) A continuation
delivered by the real delivery pass waits 16 minutes behind older
ordinary work. The task offers "move forward". The click goes through
the real `requestTaskRetry` and `resumeTaskRetry`. Result: the task has
exactly one run after the source run, the continuation; the Retry wake
is `coalesced` with `runId` = the continuation; the offer disappears;
the next delivery pass adds nothing; the real dispatcher
(`resumeQueuedRuns`) starts the continuation ahead of the older work;
after it ends, still only that one run.
- (Greptile P1, red then green) "starts nothing when the continuation
ended between showing Move it forward now and the click": the
continuation ends after the offer is read. Before the admission block
the click was accepted (`accepted: true`, a new deferred Retry that
becomes a turn). Now it returns `alreadyMovedOn`, the task has one run,
and there is no `task_retry` wake.
- "never turns a joined move-forward click into a new run after the
continuation it joined ends": the click joins the queued continuation.
The run is then cancelled through the real `cancelRun` release. Two
rounds of `resumeExecutionWaitComments`, `resumeTaskRetry`, the delivery
pass and `resumeQueuedRuns` follow. The joined wake stays `coalesced` on
the ended run. No run comes from the click, and the task's later runs
equal those of an identical task with no click (the system's own
`issue_assignment_recovery`).
- UI hook (`useTaskRetry.test.tsx`): the click sends `retryAction:
"move_forward"`, and the `alreadyMovedOn` outcome shows the plain
message.
- (a) Under 15 minutes: no Retry. Control in the herd file: without a
click the older work starts first.
- (b, presentation) Queued 15, 45 and 110 minutes: `working`, `canRetry:
true`, `retryAction: "move_forward"`.
- (c) No Retry while the continuation waits for delivery (16, 45, 110
minutes), when it is running, when a person already joined it, for a
queued run that is not a recorded continuation, and a paused agent still
asks the person to resume.
- UI (`ExecutionBlockerNotice.test.tsx`): plain wording with one "Move
it forward now" button that calls Retry; no button for a read-only user;
nothing shown for a plain `working` state.
- (d) The #21200, #21264, #21320 and #21353 guard tests pass unchanged,
including "follows the new run once a recorded continuation is
delivered" (a just-queued continuation shows no Retry).
- `node scripts/typecheck-layers.mjs` in `server/`, `tsc --noEmit` in
`packages/shared`, `pnpm run typecheck` in `ui/`: clean.
- Not related to this change:
`src/services/fresh-continuation-fallback.test.ts` has one failure
("starts the continuation while the unreachable old lease is active", a
deadlock in the `afterEach` truncate) that also fails on master
18fb6ee571 without this change.

## Risks

- Moving a waiting continuation forward lets it start before other
queued work of the same agent. That is the purpose, and it is the
ranking #21234 already gives any run a person joined.
- No Retry is offered while a continuation is still waiting for
delivery. The delivery pacing admits a waiting continuation into the
queue within minutes, and a fresh interruption at once (#21354), so the
offer appears after the continuation reaches the queue. For the recovery
owner (not changed here): offering Retry before delivery would need the
Retry admission to hold or deliver the pending continuation itself under
the task lock, instead of starting its own run beside it.
- Admission changes only for a request that carries `retryAction:
"move_forward"`. Every other Retry is unchanged. No schema, migration or
dispatch change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:03:23 -06:00
DottaandClaude Opus 5.5 681af53fa6 fix(runner): ask once more for a text answer when a turn that did tool work ends empty twice (#21359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner drives OpenCode turns and decides when a turn has a final
answer
> - A turn can do real tool work and then end with no text and no
completion call
> - The driver asks once for a final response. That request can also
come back empty, and then the run fails
> - The person then sees a failed run, although the requested work was
saved
> - This pull request adds one bounded request, text answer first, for a
turn that already did tool work
> - The benefit is that finished work gets a reply instead of a failed
run

## Linked Issues or Issue Description

**What happened?**
An agent saved a requested document with a tool. Its next response had
no text and no completion call. The driver asked once for a final
response, and that response was empty too. The run failed with
`opencode_empty_response` and showed a failure message to the person.

**Expected behavior**
When the work is done, the person gets a short final answer. An empty
response is a recoverable state, not a hard failure.

**Steps to reproduce**
1. Use an OpenCode agent on an OpenRouter model.
2. Ask it to write and save a plan document.
3. Let the provider host return an empty response after the save, and
again for the correction.
4. See the run fail although the document exists.

**Paperclip version or commit**
`master` at ff971149c2.

**Agent adapter(s) involved**
OpenCode through the native Runner.

**Additional context**
The model answered with only a completion tool call, and the serving
host returned no text and no tool call. The correction asked for the
same tool call, so it was lost again. The turn had no other recovery.

## What Changed

- `OpenCodeServerDriver`: when a turn already did tool work and the
correction comes back empty, the driver sends one more request. That
request asks for a short plain-text answer first and the completion call
after it. The text does not depend on tool-call delivery, so a chat
reply arrives even if the host drops the call again. An execution task
still gets its completion call.
- A turn makes at most two correction requests. A turn without tool work
keeps the existing single correction.
- Only tool work that the active turn reports counts. A late tool update
from an earlier turn does not earn an extra request.
- The final failure message now says how many corrections ran.
- The driver emits a separate diagnostic,
`opencode_empty_response_text_correction`, for the second request.
- `empty-response.test.ts`: two new cases from the recorded failure. One
case answers on the plain-text request and completes. One case stays
empty and fails after exactly three prompts. One case sends a late tool
update from an earlier turn and keeps the single correction.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/empty-response.test.ts
src/drivers/opencode/interrupt-empty-response.test.ts
src/drivers/opencode/interrupt-pending-tool-call.test.ts
src/drivers/opencode/output-limit.test.ts` passes. All 19 OpenCode
driver test files pass (241 tests).
- Red/green: with the driver change removed, `tool-work-text` and
`tool-work-empty` fail. With a gate that counts any tool update,
`stale-tool-work` fails. With the change, both pass.
- The existing `empty-again` case (no tool work) still fails after two
prompts, so the bound is unchanged for that case.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.

Latency:
- The change runs only after an empty response. It adds no work before
the turn starts and no work on a turn with a normal answer. Boot time
and time to first token do not change.
- On the recovery path, it adds one short model request. In the recorded
failure, one correction request took about 2.4 s.

## Risks

- Low risk. The extra request is bounded to one per turn and only after
tool work.
- In a chat, a text answer without a completion call uses the existing
chat-reply path.
- An execution task needs its completion call. If the host drops that
call again, the run fails as it did before this change, and the existing
failed-turn continuation resumes it. This change does not make that case
worse.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:57:33 -06:00
DottaandClaude Opus 5.5 9fbfd93c9b fix(runner): record an OpenCode call's serving host when the call ends (#21358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode runs that use OpenRouter record which host served each
model call in the run log
> - Operators use this host to compare quality and failures between
OpenRouter hosts
> - About one in three of the first model calls in a turn had no
recorded host, while later calls had a host about 93% of the time
> - The provider proxy cleared the recorded host when each request
started, and this clear raced the usage report of the step before
> - This pull request records a call's host when the call ends, not when
the next call starts
> - The benefit is that each step's usage names the host that served it,
so host comparisons are complete

## Linked Issues or Issue Description

**What happened?**

In an evaluation cohort of OpenCode runs through OpenRouter, 128 of 356
first usage events in a run (36%) had no `usage.upstreamProvider`. For
later usage events the rate was 168 of 2,362 (7%).

Root cause:
- OpenCode 1.18.34 sends a step's final token count only 14 to 20 ms
before it sends the next model request. We measured this three times
with the real OpenCode binary and a local fake provider.
- The provider proxy cleared the recorded host when each request
started. The driver reads OpenCode's event stream. When the next request
started before the driver handled the token event, the clear won, and
the step's usage had no host.
- If the next call's first chunk had arrived first, the step's usage
would show the next call's host.
- The first call of a turn lost the race most often, because the driver
is busiest at turn start.

**Expected behavior**

Each step's usage names the host that served that step's model call. A
call whose host the proxy cannot read carries no host. A call never
inherits another call's host.

**Steps to reproduce**

1. Start an OpenCode session through the OpenRouter provider proxy.
2. Complete one model call that names a host.
3. Start a second model call and keep its response open.
4. Let OpenCode report the first step's usage.
5. See that the usage has no `upstreamProvider`.

**Paperclip version or commit**

origin/master ff971149c2.

**Deployment mode**

Any deployment that runs OpenCode with an OpenRouter model.

**Agent adapter(s) involved**

OpenCode (Paperclip Runner OpenCode server driver).

## What Changed

- `opencode-server-driver.ts`: the provider proxy keeps the host it
reads for each call and records it when that call's response ends. It no
longer clears the recorded host when a request starts. A call that ends
without a readable host (skipped, non-2xx, redirect, or failed) records
no host.
- `opencode-server-driver.test.ts`: a new red/green test completes one
call. Then it starts a second call that has already sent its first chunk
with another host, and keeps it open. The test checks that the turn's
usage names the first call's host. Then it ends the second call and
checks that the next turn names the second call's host.
- `doc/run-log-events.md`: the "OpenRouter serving host" section now
says the host is recorded when a call ends, and why.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts
src/drivers/opencode/upstream-served.test.ts` (91 and 8 passed).
- Red/green: without the source change, the new test "keeps a finished
call's host on its usage while OpenCode's next model call is already in
flight" fails. The first step's usage names the second call's host. With
the second call's first chunk held back, the old code instead gave a
usage with no `upstreamProvider`, which is the symptom we saw in the
cohort. With the change, the test passes.
- The existing test "does not carry an earlier call's host onto a call
the tap skips, or into another run" still passes.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Latency: the change adds no `await` on the request path. It removes
one synchronous callback before the upstream request and adds one after
the response ends. The tap still forwards each chunk before it reads it,
so time to first token and boot time do not change.

## Risks

- Low risk. The change only affects run-log fields (`upstreamProvider`,
`providerRequestId`). These fields stay in the instance database and are
not sent to Paperclip Telemetry or OpenTelemetry.
- A small race remains if the driver handles a step's usage after the
whole next call has ended. That window is now one full model call
instead of about 15 ms.
- If two model calls run at the same time in one session, the usage
names the call that ended last. This was also true before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, code execution). It did the
investigation, the change, the tests, and this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:15:42 -06:00