Commit Graph
516 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 594f9d3ec2 perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee

## Linked Issues or Issue Description

**What happened?**

A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.

**Expected behavior**

Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.

**Steps to reproduce**

1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.

`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.

**Paperclip version or commit**

`master` at 2659ff1d22.

## What Changed

- `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits
for the write in progress. A newer flush for the same run replaces that
snapshot and returns the same promise. A flush for a different run (a
warm handoff) still queues its own write.
- New tests in `durable-prp-control-plane.test.ts`:
- 40 commands queued behind a slow write produce one more write, not 40.
Each caller resolves only after the stored records contain its command.
The stored records decode to the last written checkpoint.
- When the combined write fails, every caller that waited on it rejects,
and the next flush writes a full reset that contains the commands.
- New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained
heap per run, checkpoint writes, and whether finished control planes are
collected.

Why this is safe:

- A waiting snapshot has not started its write. No caller has received a
result for it, so no acknowledgement or command send depends on it.
- The replacement is captured from a later state of the same store. It
contains every change that the replaced snapshot contained.
- Every caller of the replaced snapshot waits on the same promise. The
promise resolves only after the newer snapshot is durable. A failure
rejects all of them, and the existing retry writes a full reset.
- Writes stay on one chain, in order. The server fence, the encryption,
and the stored checkpoint format do not change.
- A flush for a different run ID never replaces a waiting snapshot, so
the earlier run still gets its last state.

## Verification

- Red/green: the two new tests fail on `master` (42 writes instead of 3;
the waiting callers resolve instead of rejecting) and pass with the fix.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
in `packages/paperclip-runner`: 219 of 219 pass.
- `node scripts/check-module-boundaries.mjs` passes.
- Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s
writes, 300 events of 2 KB already in each run:

| | `master` | this PR |
| --- | --- | --- |
| Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB
|
| Retained heap per idle live run | 1,296 KB | 1,415 KB |
| Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) |
| New events committed per run in 30 s | 4 | 30 |
| Control planes collected after stop | 8 of 8 | 8 of 8 |

- With commands at 0 per second, `master` has no backlog: an inbound
event waits for its own checkpoint before the next frame is read. The
backlog comes from flushes that do not wait.

## Risks

- Behavioral shift: when a combined write fails, callers whose own
earlier revision would have been written alone now see that failure. A
checkpoint failure already disconnects the runner and the next flush
writes a full reset, so recovery is the same.
- Fewer intermediate states reach the database. A crash can only lose a
state that no caller has seen as saved, which is the same as before.
- Not covered by a test: a warm handoff whose run ID changes while a
snapshot waits. The code does not combine those; the existing
warm-transition tests pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:08:15 -06:00
DottaandClaude Opus 5.5 681af53fa6 fix(runner): ask once more for a text answer when a turn that did tool work ends empty twice (#21359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner drives OpenCode turns and decides when a turn has a final
answer
> - A turn can do real tool work and then end with no text and no
completion call
> - The driver asks once for a final response. That request can also
come back empty, and then the run fails
> - The person then sees a failed run, although the requested work was
saved
> - This pull request adds one bounded request, text answer first, for a
turn that already did tool work
> - The benefit is that finished work gets a reply instead of a failed
run

## Linked Issues or Issue Description

**What happened?**
An agent saved a requested document with a tool. Its next response had
no text and no completion call. The driver asked once for a final
response, and that response was empty too. The run failed with
`opencode_empty_response` and showed a failure message to the person.

**Expected behavior**
When the work is done, the person gets a short final answer. An empty
response is a recoverable state, not a hard failure.

**Steps to reproduce**
1. Use an OpenCode agent on an OpenRouter model.
2. Ask it to write and save a plan document.
3. Let the provider host return an empty response after the save, and
again for the correction.
4. See the run fail although the document exists.

**Paperclip version or commit**
`master` at ff971149c2.

**Agent adapter(s) involved**
OpenCode through the native Runner.

**Additional context**
The model answered with only a completion tool call, and the serving
host returned no text and no tool call. The correction asked for the
same tool call, so it was lost again. The turn had no other recovery.

## What Changed

- `OpenCodeServerDriver`: when a turn already did tool work and the
correction comes back empty, the driver sends one more request. That
request asks for a short plain-text answer first and the completion call
after it. The text does not depend on tool-call delivery, so a chat
reply arrives even if the host drops the call again. An execution task
still gets its completion call.
- A turn makes at most two correction requests. A turn without tool work
keeps the existing single correction.
- Only tool work that the active turn reports counts. A late tool update
from an earlier turn does not earn an extra request.
- The final failure message now says how many corrections ran.
- The driver emits a separate diagnostic,
`opencode_empty_response_text_correction`, for the second request.
- `empty-response.test.ts`: two new cases from the recorded failure. One
case answers on the plain-text request and completes. One case stays
empty and fails after exactly three prompts. One case sends a late tool
update from an earlier turn and keeps the single correction.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/empty-response.test.ts
src/drivers/opencode/interrupt-empty-response.test.ts
src/drivers/opencode/interrupt-pending-tool-call.test.ts
src/drivers/opencode/output-limit.test.ts` passes. All 19 OpenCode
driver test files pass (241 tests).
- Red/green: with the driver change removed, `tool-work-text` and
`tool-work-empty` fail. With a gate that counts any tool update,
`stale-tool-work` fails. With the change, both pass.
- The existing `empty-again` case (no tool work) still fails after two
prompts, so the bound is unchanged for that case.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.

Latency:
- The change runs only after an empty response. It adds no work before
the turn starts and no work on a turn with a normal answer. Boot time
and time to first token do not change.
- On the recovery path, it adds one short model request. In the recorded
failure, one correction request took about 2.4 s.

## Risks

- Low risk. The extra request is bounded to one per turn and only after
tool work.
- In a chat, a text answer without a completion call uses the existing
chat-reply path.
- An execution task needs its completion call. If the host drops that
call again, the run fails as it did before this change, and the existing
failed-turn continuation resumes it. This change does not make that case
worse.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:57:33 -06:00
DottaandClaude Opus 5.5 9fbfd93c9b fix(runner): record an OpenCode call's serving host when the call ends (#21358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode runs that use OpenRouter record which host served each
model call in the run log
> - Operators use this host to compare quality and failures between
OpenRouter hosts
> - About one in three of the first model calls in a turn had no
recorded host, while later calls had a host about 93% of the time
> - The provider proxy cleared the recorded host when each request
started, and this clear raced the usage report of the step before
> - This pull request records a call's host when the call ends, not when
the next call starts
> - The benefit is that each step's usage names the host that served it,
so host comparisons are complete

## Linked Issues or Issue Description

**What happened?**

In an evaluation cohort of OpenCode runs through OpenRouter, 128 of 356
first usage events in a run (36%) had no `usage.upstreamProvider`. For
later usage events the rate was 168 of 2,362 (7%).

Root cause:
- OpenCode 1.18.34 sends a step's final token count only 14 to 20 ms
before it sends the next model request. We measured this three times
with the real OpenCode binary and a local fake provider.
- The provider proxy cleared the recorded host when each request
started. The driver reads OpenCode's event stream. When the next request
started before the driver handled the token event, the clear won, and
the step's usage had no host.
- If the next call's first chunk had arrived first, the step's usage
would show the next call's host.
- The first call of a turn lost the race most often, because the driver
is busiest at turn start.

**Expected behavior**

Each step's usage names the host that served that step's model call. A
call whose host the proxy cannot read carries no host. A call never
inherits another call's host.

**Steps to reproduce**

1. Start an OpenCode session through the OpenRouter provider proxy.
2. Complete one model call that names a host.
3. Start a second model call and keep its response open.
4. Let OpenCode report the first step's usage.
5. See that the usage has no `upstreamProvider`.

**Paperclip version or commit**

origin/master ff971149c2.

**Deployment mode**

Any deployment that runs OpenCode with an OpenRouter model.

**Agent adapter(s) involved**

OpenCode (Paperclip Runner OpenCode server driver).

## What Changed

- `opencode-server-driver.ts`: the provider proxy keeps the host it
reads for each call and records it when that call's response ends. It no
longer clears the recorded host when a request starts. A call that ends
without a readable host (skipped, non-2xx, redirect, or failed) records
no host.
- `opencode-server-driver.test.ts`: a new red/green test completes one
call. Then it starts a second call that has already sent its first chunk
with another host, and keeps it open. The test checks that the turn's
usage names the first call's host. Then it ends the second call and
checks that the next turn names the second call's host.
- `doc/run-log-events.md`: the "OpenRouter serving host" section now
says the host is recorded when a call ends, and why.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts
src/drivers/opencode/upstream-served.test.ts` (91 and 8 passed).
- Red/green: without the source change, the new test "keeps a finished
call's host on its usage while OpenCode's next model call is already in
flight" fails. The first step's usage names the second call's host. With
the second call's first chunk held back, the old code instead gave a
usage with no `upstreamProvider`, which is the symptom we saw in the
cohort. With the change, the test passes.
- The existing test "does not carry an earlier call's host onto a call
the tap skips, or into another run" still passes.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Latency: the change adds no `await` on the request path. It removes
one synchronous callback before the upstream request and adds one after
the response ends. The tap still forwards each chunk before it reads it,
so time to first token and boot time do not change.

## Risks

- Low risk. The change only affects run-log fields (`upstreamProvider`,
`providerRequestId`). These fields stay in the instance database and are
not sent to Paperclip Telemetry or OpenTelemetry.
- A small race remains if the driver handles a step's usage after the
whole next call has ended. That window is now one full model call
instead of about 15 ms.
- If two model calls run at the same time in one session, the usage
names the call that ended last. This was also true before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, code execution). It did the
investigation, the change, the tests, and this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:15:42 -06:00
DottaandClaude Opus 5.5 3fc2198759 fix(runner): treat tool-call markup in OpenCode text as an unexecuted call (#21361)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner drives OpenCode and turns its event stream into
the run's progress, reply, and completion result
> - Some model hosts return a tool call as raw tool-call syntax inside
the assistant text, not as a structured tool call
> - The OpenCode driver streamed that syntax as progress and accepted it
as the final answer, so the run succeeded with markup as its reply and
the intended tool call never ran
> - This pull request treats leaked tool-call markup as an unexecuted,
malformed tool call
> - The benefit is that the person never sees raw call syntax as a
reply, and the agent gets one bounded correction to make the call
properly

## Linked Issues or Issue Description

**What happened?**
An agent run that uses an OpenRouter model can end with raw tool-call
markup as its reply. The model meant to call the completion tool. The
host put the call into the message text as DeepSeek DSML markup (for
example `<|DSML| invoke name="...">`). The runner accepted that text as
the final answer and settled the run as succeeded. The completion call
did not run, so follow-up automation repeated the work.

**Steps to reproduce**
1. Run an OpenCode agent with a DeepSeek model through an OpenRouter
host that leaks tool-call syntax into text.
2. Let the model end its turn with a completion call that the host
returns as text.
3. Look at the run reply.

**Expected behavior**
The runner does not show the markup and does not accept it as the reply.
It tells the model that the tool did not run and asks it once to call
the tool through the tool interface.

**Paperclip version or commit**
`master` at the time of this pull request.

**Additional context**
Today the markup streams as progress and becomes the run's final reply.
The run succeeds without its completion call.

## What Changed

- Added `tool-call-markup.ts` in the OpenCode driver. It detects leaked
tool-call markup in one linear scan: DSML and DeepSeek call-token
sentinels anywhere, and `<function_calls>` / `<tool_call>` wrappers only
at a line start with a call body. It ignores fenced code blocks.
- The driver holds markup (and a trailing fragment that can still become
markup) out of streamed progress text.
- A completed text part that contains markup is not a final-answer
candidate. The driver emits the `opencode_tool_call_markup_discarded`
diagnostic and records the position, so earlier text cannot become the
reply either.
- The turn then goes through the existing one-shot empty-response
correction. The correction text says the tool did not run and asks the
model to call tools only through the tool interface.
- Regression tests: detection cases (recorded DSML shape, other
families, prose and code-fence negatives, streaming monotonicity) and a
driver case that replays the recorded markup.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/`
passes (20 files, 263 tests).
- Red/green: the new `tool-call-markup` case in `empty-response.test.ts`
fails on the unchanged driver (the markup is selected as the final
reply) and passes with this change.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.
- Cost: the scan adds about 22 µs for each streamed update of a 20 KB
answer with `<` on every line (43 ms over 1,960 updates), and 0.05 ms
for each completed part. It adds no waits and does not touch boot or the
start of a turn.

## Risks

- Low risk. A real answer that puts DSML sentinels or a
`<function_calls>`/`<tool_call>` wrapper with a call body outside a code
fence is now treated as an unexecuted call. The correction then asks
once for a proper answer.
- If the model leaks markup again after the correction, the existing
empty-response failure path applies.
- No schema, API, or saved run-state change.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:00:29 -06:00
DottaandClaude Opus 5.5 cb94108484 fix(native): finish the run from a provider turn a timed-out start left running (#21338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run talks to a remote Runner through a controller. A
deploy or a same-run retry can reattach a new controller to the live
Runner.
> - A slow turn start can make the first controller give up with
"runnerd did not report the provider turn identity". Codex still runs
that turn.
> - The retry reattaches the Runner and adopts that turn, because the
Runner accepted it for this run's latest turn.start.
> - If the adopted turn finishes while recovery still runs, the saved
checkpoint has no active turn. The runtime then tries to start a fresh
turn, and the run fails with "session cannot start another turn". The
Runner is lost and the task needs a Retry.
> - This pull request makes the runtime treat a terminal that the
recovered session newly settled as the adopted provider terminal when
the saved checkpoint bound no turn.
> - The benefit is that a turn which a timed-out start left running
finishes the same run on the reattached Runner, with its session
context, and with no second turn/start.

## Linked Issues or Issue Description

**What happened?**
On staging, Codex runs on a remote Runner lost the Runner after a
same-run reattachment. Each chain started with a turn.start that timed
out ("runnerd did not report the provider turn identity") while Codex
ran the turn anyway. After the reattachment, the run failed with a
protocol integrity fault (thread_binding_mismatch,
tool_binding_mismatch, or native_event_replay_conflict "Provider
authoritative notification item/started did not name its execution
owner"). The task then fell back to a fresh session and lost its
context. A local reproduction with the real runnerd and the fake Codex
app server shows the same trigger also fails when the abandoned turn
finishes during the reattachment: the runtime tries a new turn and fails
with "session cannot start another turn".

**Expected behavior**
A turn that a timed-out turn.start left running belongs to the same run.
The reattached run adopts it, binds its late notifications, and finishes
from its outcome. Recovery does not kill the Runner and does not need a
manual Retry. A real cross-run or cross-task binding violation still
fails closed.

**Steps to reproduce**
1. Start a native Codex run on a remote Runner whose provider is slow to
emit turn/started after it accepts turn/start.
2. Let the controller give up with "runnerd did not report the provider
turn identity" while the Runner stays alive.
3. Let the same-run retry reattach the Runner.
4. Let the provider turn start and finish after the reattachment.
5. See the run fail instead of finishing from the adopted turn.

**Paperclip version or commit**
master at 22c9300dac (after #21321 and #21329)

**Deployment mode**
Paperclip Cloud staging, remote Runner in a Daytona sandbox

**Agent adapter(s) involved**
Codex (codex_app_server through paperclip-runnerd)

## What Changed

- `native-session-runtime.ts`: a terminal that the recovered session
newly settled now counts as the adopted provider terminal when the saved
checkpoint had no active turn. The runtime consumes that terminal
instead of starting a fresh turn.
- New `runnerd-abandoned-turn-start-reattach.test.ts` with the real
runnerd and the fake Codex app server. Each attempt runs in a fresh
module graph, as each attempt runs in its own executor process in
production. The first attempt times out on turn.start against an
unreachable remote Runner and leaves it alive.
- Red/green: the provider turn starts and finishes only after the
reattachment. The run now succeeds from `provider-turn-1` with one
turn/start and no duplicate Runner launch. Before the fix it failed with
"session cannot start another turn".
- Guard: output the abandoned turn reported while no controller was
attached binds to the reattached run's adopted turn with no
`session.failed`, and the Runner stays alive.

## Verification

- `npx vitest run
src/live/runnerd-abandoned-turn-start-reattach.test.ts` in
`packages/paperclip-runner`: red without the runtime change ("session
cannot start another turn"), green with it.
- `npx vitest run src/native-session-runtime.test.ts
src/live/runnerd-restart-continuation.test.ts
src/live/runnerd-abandoned-turn-start-reattach.test.ts
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-cut-turn-start.test.ts
src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts`: 7 files,
386 tests passed.
- `tsc -p tsconfig.test.json --noEmit` in `packages/paperclip-runner`:
no new errors (the existing `src/eval/workflow-harness.ts` errors need
the eval kernel build).

## Risks

- The runtime now adopts a newly settled terminal when the saved
checkpoint bound no turn. The driver records a turn terminal only for a
turn that it bound as active (the persisted active turn, the
uncheckpointed disposition turn, or this run's own accepted turn). A
turn of another run or task is never bound, so it still fails closed in
the driver.
- The staging chains failed with binding-mismatch codes. The local
reproduction reaches the same trigger and the adoption path, but the
fake provider cannot hold a live turn while it answers other requests,
so the exact staging codes did not reproduce locally. Late items that
reach a reattached controller before it knows the provider turn identity
can still fail closed. That path needs a separate fix if it recurs after
this change.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5) in Claude Code, with tool use
(shell, file edits, browser reads of staging run events).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21321, #21329

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:43:41 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
DottaandClaude Opus 5.5 d9a4bb23fe fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through the Paperclip runner, and the server-side
transport starts each provider turn
> - After Codex accepts a turn, the transport waits a fixed 30 s for
that turn's `turn/started` before it reports the turn identity
> - Codex starts its MCP servers before it starts the turn, so a healthy
cold start in a loaded sandbox often takes longer than 30 s
> - The transport then failed a live turn with `runnerd did not report
the provider turn identity`, and the abandoned turn's late notifications
broke the reattachment (`did not name its execution owner`,
`runner_harness_state_mismatch: resumed_not_suspended`)
> - This pull request sizes the start window to the measured cold start
and extends it while the provider keeps reporting startup progress
> - The benefit is that slow but healthy Codex starts succeed on the
first attempt, and a dead start still fails at a bound

## Linked Issues or Issue Description

**What happened?**

On staging, Codex runs on Daytona failed at turn start with
`native_session_interrupted: runnerd did not report the provider turn
identity`. The run log shows the same sequence each time. The runner
accepts the turn. Codex then reports startup progress (a bubblewrap
config warning, `account/updated`, and `mcpServer/startupStatus/updated`
failures for an MCP server that is not logged in) for about 30 s. The
transport gives up 30 s after acceptance. A few seconds later, Codex
sends the turn's `turn/started`, which now arrives at a closed turn
(`protocol_failure:turn_binding_mismatch`). The same-run reattachment
then hits that orphaned turn. It fails with `Provider authoritative
notification item/started did not name its execution owner.` or
`runner_harness_state_mismatch: resumed_not_suspended`.

Of 198 recent runs in the staging company, 52 contain this turn identity
failure. In 68 successful starts, submit to `turn/started` took p50 25
s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits
near the median.

**Expected behavior**

A slow but healthy Codex start succeeds. A start that makes no progress,
whose Runner exits, or that the runner fails still fails at a bound and
recovers through the normal same-run resume.

**Steps to reproduce**

1. Start a Codex turn whose provider accepts the turn but sends
`turn/started` after the 30 s default window (on staging: several MCP
servers and a loaded sandbox).
2. Let the provider report startup notices during the wait, and let the
runner not prove the accepted turn live in time (on staging no turn was
adopted).
3. See `runnerd did not report the provider turn identity`, then the
reattachment conflict.

The new tests reproduce this with the fake Codex app server
(`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a
runner snapshot that cannot prove liveness.

**Paperclip version or commit**

Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR:
`origin/master`.

**Deployment mode**

Authenticated, public, managed cloud (staging). Remote runner on
Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex provider (`gpt-6.1-sol`).

## What Changed

- `runnerd-codex-transport.ts`: the default wait for an accepted turn's
`turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit
`turnStartTimeoutMs` keeps its old bound.
- The wait restarts its window each time the runner commits a provider
notice, or a runner diagnostic that names the accepted turn. The total
wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can
lower it). A provider relaunch after the accepted `turn.start` does not
extend it.
- The adoption check (runner snapshot proves the accepted turn live) now
runs before each window end, not only before the first one. A Runner
exit and a runner failure of the pending turn still fail at once.
- `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends
`mcpServer/startupStatus/updated` failures while it delays
`turn/started`.
- `doc/execution-semantics.md`: documents the start window, the progress
rule and the staging measurements.

## Verification

- Red on `origin/master` transport (with the new fake flag): `keeps
waiting for an accepted Codex turn while its startup keeps reporting
progress` failed after 4.6 s, and `gives a silent Codex cold start the
measured default start window` failed after 30.9 s. Both failed with
`runnerd did not report the provider turn identity`.
- Green: `npx vitest run
src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This
includes a ceiling test: provider notices every 1 s and no
`turn/started` fail at an 8 s test ceiling, and the test goes red with
the ceiling cap removed. It also includes the existing adoption,
no-adoption-within-bound and runner-failure tests, and a new test that a
silent start still fails within its bound.
- `npx vitest run` on the adoption, cut-turn-start, stall,
turn-start-timeout-retention, remote-retention, restart-continuation and
controller-outage transport suites passes 48/48.
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392
pass. The one failure (`stops a lost runnerd's provider and escaped tool
job from its lineage journal (journal=true)`, 1 s, process-lineage
timing under full-suite load) passes when run alone, both on this branch
and on `origin/master`.
- `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check`
pass.

## Risks

- A silent dead start (no Runner exit, no provider output, no runner
failure) now fails after 90 s, not 30 s. The Runner exit and
runner-failure paths still fail at once, so this applies only to a
provider that stops without a trace.
- A provider that keeps sending notices but never starts the turn waits
at most 180 s.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #21272, #21120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:21:37 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
Nicky LeachandClaude Fable 5.2 ff91bfe2dc fix(runner): withhold Claude's self-scheduling tools from Paperclip ACPX sessions (#21276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip owns scheduling. An agent uses Paperclip tools such as
task monitors and routines to continue work later.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude Code offers its own scheduling tools to the model:
`CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. With these
tools the provider can wake itself. Paperclip does not see or govern
that work.
> - Today the Runner stops the Claude process when a turn ends, so these
tools have little effect. A later change keeps the Claude process alive
between turns.
> - This pull request removes the four tools from Claude sessions that
Paperclip starts through ACPX.
> - The benefit is that a Claude agent schedules work only through
Paperclip.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #19912 (the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

The set of tools that Claude Code offers the model in a Paperclip ACPX
session.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

A Paperclip ACPX Claude session offers the model 26 built-in tools. Four
of them are `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.

**Proposed behavior**

- The Claude settings that Paperclip writes into the session home deny
the four tools. Claude Code then does not offer them.
- Each Claude launch sets `CLAUDE_CODE_DISABLE_CRON=1`. This stops the
cron scheduler of Claude Code and removes the three cron tools.
- All other tools stay. `SendMessage`, `ListAgents`, `Workflow`, `Agent`
and `TaskStop` do not change.

**Reason and benefit**

The provider cannot schedule its own wake-ups. Paperclip tools stay the
only way to schedule work.

## What Changed

- `packages/paperclip-runner/src/drivers/acpx/permission-policy.ts`: new
constant `CLAUDE_SELF_SCHEDULING_TOOLS` with the four tool names.
- `packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts`: the
Claude `settings.json` now always has `permissions.deny` with the four
tools, in each permission mode. The `allow` rules do not change.
- `packages/paperclip-runner/src/drivers/acpx/environment.ts`:
`createSanitizedAcpxSpawnInput` sets `CLAUDE_CODE_DISABLE_CRON=1` for
each Claude launch. The host cannot override it.
- New test `claude-scheduling-tools.claude-acp.test.ts`. It prepares the
session home with the production `prepareAcpxRuntimeSandbox`. It then
runs the pinned `claude-agent-acp` and the pinned Claude Code client
against a local fake Anthropic API. It reads the tools that the client
offers the model.
- Unit tests in `environment.test.ts` and `runtime-sandbox.test.ts`. One
assertion in `runtime-host.test.ts` now expects the `deny` list.
- No change to the `claude-agent-acp` patch, the lockfile or a pinned
digest. No change to Codex.

## State saved by the previous release

- A session that the previous release saved continues with no change.
The new restriction is not part of a saved identity or digest.
- Each time Paperclip starts Claude for a session, it writes its own
`settings.json` again with the deny list and sets
`CLAUDE_CODE_DISABLE_CRON`. This includes a resumed session and a start
after an upgrade.
- A resumed session keeps its identity, its home and its transcript. It
loses the four tools from its next start.
- A Claude process that the previous release started keeps its tools
until it stops. Paperclip does not refuse it and does not classify it
differently. Its next start gets the restriction.
- No refusal code is necessary, because no saved state becomes
incompatible.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts`
- Without the product change: `2 failed` (`expected [ Array(26) ] to not
include 'CronCreate'`).
  - With the product change: `2 passed`.
- `npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts`: `4 passed (4)` files, `147
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` in `packages/paperclip-runner`: no
errors.
- Tools offered after the change, isolated home: the same 26 tools
without `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.
- `npx vitest run src/drivers/acpx/runtime-host.test.ts -t "previous
release"`: the new test "resumes a Claude session saved by the previous
release and withholds self-scheduling tools on relaunch" fails without
the product change and passes with it. The test saves a session, puts
its home into the shape of the previous release, and resumes it through
`AcpxRuntimeHost.open`.
- `npx tsc --noEmit -p tsconfig.test.json` in
`packages/paperclip-runner`: no errors.
- The fixture in `claude-scheduling-tools.claude-acp.test.ts` now stops
the process group of the provider and waits for it before it removes the
fixture directory.
- Not run: a live run with a real Claude account, and a remote Runner.
Not run: `pnpm verify` and the full suite. CI supplies that coverage.

## Risks

- **A preconfigured home still offers `ScheduleWakeup`.** In that mode
Paperclip does not write the Claude settings of the user, so the deny
list does not apply. The environment variable removes only the three
cron tools. To remove `ScheduleWakeup` there, the Runner must pass
`disallowedTools` to the session. ACPX 0.22 does not forward that
option. This is a follow-up.
- **The new test runs the real pinned Claude Code client.** It needs the
platform binary, as `runner-tool-bridge.claude-schema.test.ts` does
today. It uses no credentials and no network.
- **A prompt or skill that tells Claude to use these tools no longer
works.** No Paperclip instruction refers to them.
- The prepared harness home and remote Runners use the same code path.
They are not tested separately.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the options
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:17:24 -08:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
DottaandClaude Opus 5.5 a298ca7fbc fix(runner): prove ownership right before starting a turn in a recovered session (#21260)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner runs native agent sessions (Codex, Claude, and
others) and keeps them alive across server swaps. A new server
generation recovers the session and continues it.
> - Only the generation that owns the run may submit a turn into the
provider session. The Runner proves ownership with a fenced checkpoint
(controller boot, generation and lease, judged by the database clock).
> - On a recovered session, the startup turn and the restart
continuation took that checkpoint right after recovery, then prepared
the turn's input (attachments, queued input, steered messages), then
submitted.
> - A generation that lost the run during input preparation still
submitted a turn into the shared provider session.
> - This pull request repeats the fenced checkpoint right before
`turn.start` on those two recovered-session paths.
> - The benefit is that a server generation that no longer owns a run
cannot add a turn to its provider session. New sessions are unchanged,
so nothing is added before their first turn.

## Linked Issues or Issue Description

**What happened?**
On staging, a Codex run crossed three server swaps. After its turn
finished without a result, a turn that no generation could account for
appeared in the same provider thread. Every path that submits a turn
into an existing session is preceded by the fenced checkpoint, but the
two startup paths took it before input preparation, not right before the
submit. A generation that lost the run in that window still submitted.

**Expected behavior**
A generation submits a turn into a recovered session only if it still
owns the run at the moment of the submit.

**Steps to reproduce**
1. Recover a native session on a new server generation.
2. While it prepares the startup turn's input, fence the generation
(another generation takes the lease).
3. Before this change, the turn is still submitted. After it, the
checkpoint fails and no turn starts.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
- The restart continuation calls the fenced `checkpoint` right before
`session.startTurn`, after attachments and steered messages are
prepared.
- The startup turn on a recovered session (`recovered === true`) calls
`checkpoint` right before `session.startTurn`. A new session's normal
path is unchanged.
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: new
tests "proves it still owns a recovered session right before starting a
turn in it" (startup turn) and "proves it still owns a recovered session
right before a restart continuation" at startup and when the lost turn's
terminal arrives (`resumeInterruptedTurn`).

Turn starts into an existing session, and how each proves ownership
right before the submit:
- Startup turn on a recovered session: new checkpoint, covered by the
first test.
- Restart continuation (`startRestartContinuation`), at startup and via
`resumeInterruptedTurn`: new checkpoint, covered by the second test
(both cases).
- Continuation disposition (`requestContinuationDisposition`): already
persists the fenced checkpoint immediately before `startTurn`, with only
a snapshot read between them and no input preparation. No change.
- Compaction turn: started right after the fenced recovery checkpoint
with an empty message; there is no input preparation between them (the
only possible await is the disposition-replay read, which applies only
when a disposition turn was submitted, and then no compaction turn
starts). No change.
- Runner-only change. No change to `protocol/`, `generated/`, Rust
shapes or the ACPX patch, so the provider-pack `contractDigest` and
interface revisions are unchanged and older image packs stay compatible.
The Runner build and image content for this source change are produced
by CI (`runner-artifacts`); nothing is deployed by this PR.

## Verification

- Red-to-green: "proves it still owns a recovered session right before
starting a turn in it" fails without the startup-turn checkpoint (the
turn is submitted after the generation is fenced) and passes with it.
Both restart-continuation cases fail with the checkpoint removed from
`startRestartContinuation` and pass with it.
- Local: `cd packages/paperclip-runner && pnpm exec vitest run
src/native-session-runtime.test.ts`: 290 passed.
- Local: `node scripts/typecheck-test-programs.mjs
packages/paperclip-runner/tsconfig.test.json`: passed (the first CI run
failed on the test's attachment prompt returning `undefined`; it now
returns a string).
- **Not run locally:** one test case that needs the Rust
`paperclip-runnerd` binary, and
`src/live/runnerd-restart-continuation.test.ts` (10 cases). The Rust
binary was not built on this machine. CI must cover them.

## Risks

- One extra checkpoint round trip before `turn.start` on a recovered
session and on a restart continuation. These are recovery paths; a new
session's first turn gets no added wait, so boot and time-to-first-token
for new runs are unchanged.
- If the checkpoint fails (the generation lost the run), the turn is not
submitted. That is the intended behaviour; the owning generation
continues the run.
- No migration. No schema change. No pack contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:08:19 -06:00
DottaandClaude Opus 5.5 52817dff55 fix(runner): name missing tool fields and what the call included (#21252)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools, and the runner validates each
tool call against its input schema
> - When a call is invalid, the runner returns an "Invalid tool input"
error to the model
> - For a call that left out required fields, that error was a list of
JSON Schema messages such as `/ must have required property 'kickoff'`
> - A model sent `paperclip_create_case` with only
`workflow_document_text` for about 5 minutes, believed it had sent every
field, and never replied to the person
> - This pull request makes the error name the missing fields, the
fields the call did include, and a short example of the missing fields
> - The benefit is that a model can see its own mistake and correct the
call on the next attempt

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `cases.create-case` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), the agent called
`paperclip_create_case` about 20 times with only
`workflow_document_text`. Each call returned `Invalid tool input: / must
have required property 'idempotencyKey'; / must have required property
'caseType'; ...`. The model's reasoning said that the other fields were
being stripped. It never created the case and never replied.

**Steps to reproduce**
1. Validate `{ "workflow_document_text": "..." }` against the
`create_case` input schema.
2. Read `toolInputError(validator)`.

**Expected behavior**
The error says which required fields are missing, says that the call
included only `workflow_document_text`, and shows a short example of the
missing fields.

## What Changed

- `packages/paperclip-runner/src/drivers/tool-input-error.ts`: for
missing top-level required fields, the error now says `missing required
fields: idempotencyKey, caseType, title, kickoff.`, `This call included
only: workflow_document_text.`, and a placeholder example built from the
schema, such as
`{"idempotencyKey":"<idempotencyKey>",...,"kickoff":{"toAgentId":"<toAgentId>","phase":"<phase>","instructions":"<instructions>"}}`.
Other validation errors follow after `Also:`. Field-description guidance
is unchanged.
- `packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts` and
`runner-tool-bridge.ts`: pass the call arguments to the error builder.
Only argument names are read; values are never echoed.
- `packages/paperclip-runner/src/drivers/tool-input-error.test.ts`: a
test on the real `create_case` schema and a test that other errors still
follow the missing fields.
- The `create_case` schema is not flattened. The model also left out the
flat fields (`idempotencyKey`, `caseType`, `title`), so nesting was not
the cause, and `kickoff` is shared with `handoff_case`.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/tool-input-error.test.ts
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/opencode/opencode-server-driver.test.ts` passes (115 tests).
The two new tests fail on master.

## Risks

- Low risk. Only the error text for invalid calls changes. Validation
and dispatch are unchanged. The example is bounded to 600 characters and
the field lists to 12 names.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:06:48 -06:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Opus 5.5 ef6d347f79 fix(runner): keep reattached runners answering after a restart (#21161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry

## Linked Issues or Issue Description

**What happened?**

After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:

- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`

Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).

Root causes, proven with the real Runner and the fake Codex harness:

1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.

**Expected behavior**

A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.

**Paperclip version or commit**

`master` at 3fb35fcf13.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.

### Changed existing test

`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.

## Verification

- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.

## Risks

- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:50:43 -06:00
DottaandClaude Opus 5.5 c5e4b46d3f fix(acpx): shut down the per-port ACPX runtime when its session closes (#21151)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The experimental Runner starts ACP providers through ACPX, and each
runtime port owns one ACPX runtime
> - When the native provider dies during a turn, the port retires the
turn at once, but the turn stays queued inside the ACPX runtime
> - `port.close()` closed only the session handle, so that queued turn
wrote its final session record after close had returned
> - Any caller that removes the state directory after close races that
write; CI showed it as `ENOTEMPTY ... state/sessions` in an unrelated
test on 5 branches
> - This pull request makes close shut down the port's own ACPX runtime
after the session closes, inside the existing close bound
> - The benefit is that a closed port no longer writes into its state
directory, and the flaky test is fixed at its cause

## Linked Issues or Issue Description

**What happened?**
`packages/paperclip-runner` test `native-distribution-integrity.test.ts
> native ACPX execution closure > expires a real ACPX permission on
native child death without replaying its mutation` fails intermittently
with `Error: ENOTEMPTY: directory not empty, rmdir
'/tmp/native-acpx-test-XXXX/state/sessions'`. It failed 5 times on 5
unrelated branches in 48 hours.

**Steps to reproduce**
1. Run the test 30 times while two CPU-bound shell loops run.
2. 2 of 30 runs fail with ENOTEMPTY in `afterEach`.
3. Instrumenting the session store shows a `saveSessionOwner` write from
`finalizeRuntimeTurn` that starts after `port.close()` and `afterEach`
have begun.

**Paperclip version or commit**
`master` at the base of this branch (CI on unrelated pull requests,
2026-11-10 to 2026-11-12).

**Expected behavior**
After `port.close()` resolves, the ACPX runtime owned by that port does
no more work and writes no more state.

**Actual behavior**
The turn that native provider death retired is still queued in the ACPX
runtime manager. It finalizes after close and writes the session file
while the caller removes the directory.

## What Changed

- `acpx-runtime-adapter.ts`: after the final successful close, the port
calls `runtime.shutdown()` within `runtimeCloseTimeoutMs`. ACPX shutdown
waits for its pending turn tasks, so the abandoned turn finishes its
writes first.
- A close that succeeds late, after its bound, also starts the shutdown.
A concurrent or repeated `close()` waits for the same shutdown and gets
its failure too. A shutdown failure is reported in the existing `ACPX
runtime and provider cleanup failed` error. The shutdown attempt stays
in the retained cleanup owners.
- `acpx-runtime-adapter.test.ts`: regression tests. Close does not
resolve before runtime shutdown settles. A shutdown failure is reported
to every closer. A late close success starts the shutdown.

## Verification

- Red/green: both new tests fail on master (`expected "vi.fn()" to be
called once, but got 0 times`; `promise resolved "undefined" instead of
rejecting`) and pass with this change.
- Loop of the flaky test, 30 runs each, with 2 extra CPU-bound shell
loops on an already loaded 18-core Mac (load average about 30):
  - Before: 28 pass, 2 fail (both `ENOTEMPTY ... state/sessions`).
  - After: 30 pass, 0 fail.
- `npx vitest run src/drivers/acpx` in `packages/paperclip-runner`: 92
files, 1743 tests pass.
- `tsc --noEmit -p packages/paperclip-runner`: clean.

## Risks

- Low risk. The runtime is created per port and is not used after a
successful final close. Close can now take up to one more close bound if
the runtime's pending work is slow. A late reconciliation close never
runs after this point, because shutdown only starts when the port is
finally closed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:29:36 -06:00
DottaandClaude Opus 5.5 814fe5d20f fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox start `runnerd` in the sandbox, and
Paperclip connects to it through provider ingress and authenticates it
with a one-use bootstrap ticket
> - The ticket lives for 60 seconds, but Paperclip issued it before the
remote launcher staged its files in the sandbox
> - Under load, staging took 60 to 81 seconds, so `runnerd` presented an
expired ticket and every connection was closed. The run failed with
`runner_ingress_auth_failed`, and the task stopped behind a manual Retry
> - This pull request issues a fresh ticket right before dispatch,
retries these failures automatically when the controller checkpoint
proves no runner ever authenticated, and adds spans that attribute the
slow startup window
> - The benefit is that a slow but healthy start authenticates, a failed
start recovers on its own, and the remaining startup delay becomes
measurable

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, 9 runs of one Codex agent on Daytona failed in
about 70 minutes with `native_session_interrupted` and the message
`runner_ingress_auth_failed: Runner ingress did not complete PRP
authentication before the deadline.` In the same window only 6 runs
succeeded company-wide. The run log showed Paperclip reconnecting for 60
seconds while `runnerd` reported that its peer closed each WebSocket. In
294 remote runs with timing spans, the time between the bootstrap ticket
issue (end of `runner.transport.connect`) and the process dispatch
(start of `runner.process.launch`) decides the outcome without
exception:

- 60 s or more: 23 of 23 runs failed authentication.
- Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s.

The tasks did not recover by themselves, because the failure was
classified as ambiguous state.

**Expected behavior**

A runner that starts slowly but correctly authenticates. If ingress
authentication still fails before any runner authenticated, Paperclip
retries the start automatically. The time between the start request and
the runner connection is visible in run spans.

**Steps to reproduce**

1. Run a Codex agent on a Daytona sandbox environment that uses provider
ingress.
2. Make remote staging (`runner.runtime.stage`, inside the remote
launcher) take 60 seconds or more. Many concurrent runs of the same
agent do this on staging.
3. See that `runner.prp.authenticate` fails after 60 seconds and the run
ends with `runner_ingress_auth_failed`.
4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts`
reproduces this locally with a 75-second staging delay.

**Paperclip version or commit**

`4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`.

**Deployment mode**

Hosted (cloud), controller on Railway, sandboxes on Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`),
remote sandbox target, provider-ingress transport.

## What Changed

- `RunnerProcessLaunchSpec` and `spawnRunner` accept
`refreshBootstrapTicket`. A restart does not inherit it.
- The runnerd Codex transport passes a refresh that issues a new
bootstrap ticket and flushes the controller checkpoint. It does not do
this for a warm-transition launch.
- The remote sandbox launcher calls the refresh after `ensureArtifact`
and immediately before the launch RPC, and puts the new ticket in the
launch environment.
- `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller
checkpoint that no runner authenticated: no connections, no leases, no
used tickets, no events, and only pending commands.
- `nativeSessionFailureSourceCode` classifies the outbound ingress
startup failures as `runner_ingress_startup_failed`.
`nativeProviderRecoveryEvidence` treats it as pre-session
(`bootstrap_retry`) only when that proof holds for the exact run
identity and there is no provider session or provider event. Restart
recovery and failed-run retry accept the new code with the same rules as
similar startup codes.
- Remote startup now records `runner.control_plane.open`,
`runner.launch.prepare`, `runner.checkpoint.register` and
`stage.asset.codex_skills.verify`. The 6 to 11 minute gap between
`native.process_start_requested` and `runner.transport.connect` had no
spans before.

## Verification

- Red/green:
`packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts`
starts a real `runnerd` with a 75-second staging delay. Without the fix
it fails with `runner never connected`. With the fix it opens the
session.
- Red/green: `native-session-executor.test.ts` "launches runnerd with a
bootstrap ticket issued after remote staging". Without the fix the
launch RPC gets the stale ticket.
- Red/green: `native-controller-recovery.integration.test.ts` "retries a
fresh remote start whose launched runner never authenticated through
provider ingress" (embedded Postgres). Without the fix the failure is
`native_session_interrupted` and `ambiguous_state`.
- `durable-prp-control-plane.test.ts`: the new checkpoint proof, its
negative cases, and the restart rule for the refresh.
- `native-run-trace.test.ts`: remote startup spans are recorded without
duplicate launcher spans.
- Focused suites pass: runner package 5 files and 235 tests; server 9
files and 1,607 tests, including `native-session-executor`,
`native-runner-execution-driver`, `remote-launch-assets`,
`native-restart-recovery`, `runner-prp-outbound` and the recovery
integration tests.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- A launch now issues two bootstrap tickets. The first one is never sent
to `runnerd` and expires in 60 seconds. Bootstrap counters already allow
more than one issued credential.
- The new retry path needs a checkpoint proof that no runner
authenticated, so a run that did provider work cannot be retried this
way. The existing attempt budget still bounds the retries.
- This change does not remove the 6 to 11 minute controller-side delay
before `runner.transport.connect`. The new spans make it attributable
for a follow-up change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, code search) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:30 -06:00
DottaandClaude Opus 5.5 b7693e2076 fix(runner): decode native inputs saved under ACPX 0.13.1 (#21062)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner defines the native execution input that every
native run saves and every later run reads back
> - The ACPX upgrade to 0.22.0 (#20770) changed the qualified
`acpxVersion` in each ACPX profile
> - The input parser accepts only the current `acpxVersion`, so it
refuses every input that was saved under ACPX 0.13.1
> - Run preparation reads the previous run's saved input for the same
task, so each ACPX agent with an older task session fails before its
provider starts
> - This pull request lets the parser decode inputs saved under ACPX
0.13.1 and normalizes them to the current qualified profile
> - The benefit is that saved ACPX agents and tasks work again after the
upgrade, with no manual edit

## Linked Issues or Issue Description

Refs #20770

**What happened?**
Every run of an ACPX agent (Claude through ACPX) failed about 20 seconds
after it started. The run error was `input.provider.profile does not
match the qualified ACPX v1 profile` with the failure cause
`native_execution_input_invalid`. The run log showed no runner output,
only the preparation messages. Five runs on five different tasks failed
this way in 40 minutes. The previous successful run on one of those
tasks saved this provider profile:
`{"agent":"claude","acpxVersion":"0.13.1","agentProfileVersion":1,...}`.
New runs saved `acpxVersion` `0.22.0`. Run preparation reads the
previous run's saved input (for example, the instruction working copy
step parses the input of the task session's last run). The parser
refused `0.13.1` and the error stopped the run.

**Expected behavior**
A run on a task whose earlier run was saved under ACPX 0.13.1 prepares
and starts normally. The ACPX upgrade carries saved sessions forward.
The session identity already pins `0.13.1` for this reason
(`recovery-identity.ts`). The saved input must decode too.

**Steps to reproduce**
1. Run an ACPX agent on a task with Paperclip before #20770, so the run
saves `provider.profile.acpxVersion: "0.13.1"`.
2. Deploy a build that includes #20770.
3. Wake the agent on the same task.
4. See the run fail in preparation with `input.provider.profile does not
match the qualified ACPX v1 profile`.
5. The new unit test reproduces this with the exact saved profile shape.

**Paperclip version or commit**
master at 424e2c6488. The failure is on a deployed build at 3c058c37e6,
which includes #20770.

**Deployment mode**
Hosted cloud (staging), Daytona remote workspace, `paperclip_runner`
adapter.

**Agent adapter(s) involved**
`paperclip_runner` with provider `acpx`, ACPX agent `claude`. All ACPX
agents with inputs saved before the upgrade have the same failure.

## What Changed

- `profile-compatibility.ts`: add `isDecodableAcpxVersion`. It accepts
the current qualified ACPX version and `0.13.1`, the only ACPX version
that was qualified before.
- `native-execution.ts`: the ACPX profile check uses
`isDecodableAcpxVersion` instead of a strict equality with the current
version. The parser continues to write the current qualified version
into the parsed profile, so all later checks and digests see the current
profile.
- `native-execution.test.ts`: a regression test with the exact saved
Claude profile shape. It confirms that the saved input decodes, that the
result is the current profile, that the parse is idempotent, and that
other versions (`0.13.0`, `0.21.9`, `0.23.0`, empty, null, number) are
still refused.

## Verification

- Red: the new test failed on master with `input.provider.profile does
not match the qualified ACPX v1 profile`, the same error as the failed
runs.
- Green: `npx vitest run src/contracts/native-execution.test.ts` in
`packages/paperclip-runner`: 115 passed.
- `npx vitest run` on the ACPX `profile-compatibility`,
`cursor-cloud-candidate-profiles`, `kimi-installation`,
`pi-installation`, `copilot-installation` and `recovery-identity` tests:
140 passed, 1 skipped.
- Server: `native-execution-input.test.ts` (42 passed) and
`heartbeat/instruction-working-copy.test.ts` (14 passed).
- `tsc --noEmit` for `packages/paperclip-runner`: clean.
- `pnpm verify --explain` selects the full suite because the runner
package changed. CI supplies that coverage.

## Risks

- Low risk. The change only widens decoding from one ACPX version to two
named versions. Unknown versions are still refused.
- A decoded 0.13.1 input becomes the current qualified profile. This
matches the upgrade design: the current ACPX loads sessions that 0.13.1
wrote, and the session identity already uses 0.13.1 for all versions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser for read-only staging API reads) and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:35:32 -06:00
DottaandClaude Opus 5.5 4228bcf89c fix(opencode): never save the model's reasoning as its chat reply (#21051)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode models reply in chat through the Runner's
OpenCode driver. The driver settles one final response, and Paperclip
saves it as the reply comment
> - In chat-eval cohort 5 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), four saved replies were
the agent's reasoning, for example "Confirmed down. Now report honestly.
... I'll finish with paperclip_finish."
> - The run events show the cause. The model wrote a reasoning part,
then wrote the same text again as an ordinary text part, and ended the
step without `paperclip_finish`. The driver took that copy as the final
answer. This is not the #20999 path: no completion call happened
> - A second, related problem is in the #20999/#21025 path. When the
model writes an answer, calls `paperclip_finish`, and then writes its
own new answer, the driver can save both texts, or a status note
together with the answer
> - This pull request drops the repeated reasoning, lets the existing
empty-response correction ask for the real answer, and saves the
post-completion message alone unless it is a short line that refers back
> - The benefit is that people never see the agent's planning text or
tool names as its reply, and the eval catalog now fails when they do

## Linked Issues or Issue Description

Refs #20999
Refs #21025

**What happened?**
The saved chat reply held the agent's reasoning, not an answer. Cohort 5
cases: `connection-setup.unreachable-mcp`, `plugins.configure-plugin`,
`tasks.website-task`, `teams.install-team`. In each case the reply text
is byte-identical to the reasoning part before it, and the run result
evidence is `provider:opencode:agent-message` (no `paperclip_finish`
call). Also, `recovery.explain-stall` saved its answer twice, and
`status-cards.refresh-card` saved a "STATUS: ..." note above its answer.

**Expected behavior**
The saved reply is the model's answer to the person. It never holds
reasoning, completion tool names or report field names. A short closing
line such as "Answered above" still keeps the earlier answer.

**Steps to reproduce**
1. Use an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`.
2. Ask a chat request that the model finishes without a completion call
after a long reasoning step.
3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds
the reasoning text.

**Paperclip version or commit**
master at 98c18784c6 (cohort 5) and 3c058c37e6 (base of this PR).

## What Changed

- `OpenCodeServerDriver` records the text of each completed reasoning
part in the run. A text part that still matches the start of a completed
reasoning part does not stream as progress. A completed text part that
repeats a reasoning part exactly is not a response candidate, and the
driver emits the `opencode_reasoning_echo_discarded` diagnostic. With no
answer left, the existing empty-response correction asks the model for a
final answer with reasoning disabled.
- After the completion call, the driver keeps the prose written between
the last work tool and that call only when the later message is a
closing reference: a short line with no link of its own that points back
to it ("Answered above.", "Commands above are ready"), or one under half
that prose's length. A later message that repeats the prose is saved
once. Otherwise the later message is the reply. The completion tool
description already names the post-call message as the final response.
This replaces the `restatesEarlierReply` heuristic from #21025, which is
removed.
- Tests: a new `reasoning-echo` mode in `empty-response.test.ts` uses
the recorded `tasks.website-task` text. Two new driver tests use the
recorded `recovery.explain-stall` and `status-cards.refresh-card` texts.
The #20999 "Answered above" test and the #21025 restatement test pass
without change. After review: the reasoning-echo mode streams the copy
with a trailing newline (it must stay held back), a new driver test
keeps "The result is 42." when the later message is "Answered above.",
and two more check that a STATUS note followed by a short complete
answer saves only the answer, and a repeated short answer is saved once.
- No eval keyword gate: a draft shared check that matched completion
tool names in replies was removed after review, because chat catalog
graders must not grade free-form replies by keyword. Replies stay saved
for human review; the driver fix is what keeps reasoning out of them.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/drivers/opencode`:
17 files, 207 tests pass.
- Red/green: with the driver from `origin/master`, the `reasoning-echo`
test fails (the saved reply is the reasoning text), and both new
post-completion tests fail (two texts saved). With this change, they
pass.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-reply
tests/runner-e2e/source-sdk`: 3903 tests pass.
- Check calibration: the new unit tests cover recorded valid replies,
reworded valid replies (`paperclip_runner` adapter type,
`PAPERCLIP_API_KEY`, `paperclip_finish_task`, plain
"summary"/"evidence"), and the four recorded broken replies. Over all
193 cohort-5 conversations the check flags exactly the four broken
cases.
- Over the 30 cohort-5 turns with prose before and after
`paperclip_finish`, the new rule changes 3: `recovery.explain-stall`
(duplicate removed), `status-cards.refresh-card` ("STATUS:" note
removed) and `assistant-connections.consent-read-only` (shorter
restatement kept, same link). The pointer-line turns
(`tasks.website-without-project`, `cli-operations.choose-cli-context`)
still keep both texts.
- `npx tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- A paid chat-eval run on this change is linked in a comment below.

## Risks

- A turn that repeats its reasoning as text now takes one more model
request (the existing empty-response correction). If the corrected
response is empty again, the turn fails with `opencode_empty_response`,
as before.
- A real answer that is exactly equal to a reasoning part is dropped and
requested again. The correction recovers it.
- When a long post-completion message drops something the earlier prose
said, only the later message is saved. The completion tool contract
names that message as the final response, and the earlier prose remains
in the run transcript.
- Saved run state: no persisted run, session, checkpoint or event shape
changes. The reasoning texts are in-memory driver state per run. A run
saved by the previous release resumes without change; only a newly
settled final response differs.
- The new catalog check can fail a case that passed before. In cohort 5
it fails only the four reasoning replies.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:06:40 -06:00
DottaandClaude Fable 5.2 fd9985b571 fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes talk to a Runner through durable PRP
commands. The controller waits for each command's result
> - After a server replacement, the new controller reattached to live
Runners. For runs it reached later than about 115 s after detach,
`session.snapshot` got no answer within 30 s
> - The failure said only `provider_transport_failed: PRP command
session.snapshot timed out`. The run log could not tell whether the
Runner was disconnected, whether the command waited behind older
commands, or when the Runner last sent an event
> - This pull request adds that evidence to the timeout message. It
names types, counts and durations only, never payloads
> - The benefit is that the next upgrade's failures explain the stall,
so the runner-side cause can be fixed

## Linked Issues or Issue Description

Refs #21053, #21052, #21039

**What happened?**
At the 07:10Z staging swap, 9 remote native runs that the new controller
reached 114–152 s after detach failed. Each recovery read ended with
`PRP command session.snapshot timed out`. The transport had
authenticated, and in some runs the same Runner process was verified.
The Runner journal recorded no diagnostic other than "WebSocket peer
closed the connection". Runs reached within 39–116 s succeeded.

**What did you expect to happen?**
A command timeout should say what the command waited behind, so the
cause can be found from the run log.

**Steps to reproduce**
1. Start a remote native run and replace the server while a provider
turn is active.
2. Let the new controller reach the Runner more than about two minutes
after detach.
3. Read the run log. The failure says only "timed out".

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
`waitForRunnerCommand` accepts an optional `describeStall`. When a
command times out, the message ends with a bounded description, for
example `(runner connected; pending ahead 1 [semantic_tool.result]
oldest 188s; last runner event seq 152 181s ago)`. The description
covers whether a Runner is connected or how long it has been
disconnected, the pending commands ahead of this one (types and count,
up to 6 types, and the age of the oldest), and the last committed Runner
event's sequence and age. A failing or empty description keeps the plain
message.
- The transport passes this description for every command it waits on.
The last event's age comes from the PRP payload's `emittedAt`.
- `server/src/services/native-runtime/native-session-executor.ts`: the
startup `session.open` timeout classifier accepts the optional
description, so a startup timeout stays `provider_session_open_timeout`
and keeps its bounded same-run retry. A regression case covers each
startup code with the suffix.
- Tests: the exact message for the staging shape, the plain message when
there is nothing to describe or the description throws, and a
disconnected Runner with no events.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets
no answer"` passes 3 of 3. These tests fail on `origin/master`, which
has no `describeRunnerCommandStall` export and no description in the
message.
- The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The
one failure (`stops a lost runnerd's provider and escaped tool job from
its lineage journal`) is a process-timing test that passes alone and
does not touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Message matchers: the only matcher anchored on the end of the timeout
text was the startup `session.open` classifier. It now accepts the
suffix. `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"classifies a startup session.open timeout"` passes. Without the
classifier change it fails, because the suffix turns the timeout into
`native_session_interrupted`.

## Risks

- Low. Only the text of a timeout message changes, by appending text. No
control flow, persisted state or Runner behavior changes.
- The description holds command types, counts and durations only. It
holds no payloads, identifiers or provider text.
- No wait is added before a turn start.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 03:00:32 -06:00
DottaandClaude Fable 5.2 8e956da1cf fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner. A
server replacement detaches the controller and leaves the Runner alive.
The new server reattaches to it and continues the same turn
> - On staging, about half of the in-flight runs at each upgrade lost
their turn. The new controller reattached to the live Runner (same
process verified), its recovery read timed out (`PRP command
session.snapshot timed out`), and the failure cleanup killed the Runner
> - The next same-run reattachment then found `process_not_running`.
#21039 continues these runs in a fresh session, but the in-flight turn
is lost and its outcome is unknown
> - The cause of the kill is in the Codex driver's `recoverSession`: its
cleanup closed the transport without retention. For a reattached Runner
that means SIGKILL after the close deadline. #21023 already keeps the
Runner for a failed open session, but not for a failed recovery
> - This pull request keeps the live Runner when a recovery fails or is
aborted on its way to it. Identity refusals and protocol integrity
faults still stop it
> - The benefit is that a slow reattachment no longer destroys a turn
that was still running

## Linked Issues or Issue Description

Refs #21023, #20923, #21039, #21052

**What happened?**
Staging replaced its server at about 07:10Z with 17 remote native runs
in flight. For each, the old server logged "Server shutdown suspended
native controller ownership without cancelling provider work". The new
server reattached. Example run: the Runner process (same pid and start
time) was verified at 07:12:37 and the transport authenticated at
07:12:32. At 07:13:28 the recovery failed with
`native_session_recovery_failed: ... PRP command session.snapshot timed
out`, and a same-run resume was saved. At 07:14:12 the same-run
reattachment found `process_verification` / `process_not_running`, with
the Runner journal still `ready` and no exit record from the Runner
itself. The run failed with `runner_remote_process_identity_unavailable`
/ `native_continuation_requires_reconciliation`. 90 of 93 such failures
in two hours of upgrades had this sequence.

**What did you expect to happen?**
A recovery that times out against a live Runner is a transient failure.
The same-run reattachment should find the Runner alive and adopt it
again. Killing the Runner is only for a deliberate stop, an identity
refusal, or an integrity fault.

**Root cause**

`packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`
`recoverSession`: the `catch` block and the abort handler call
`bootstrapCancellation(...).close()`, which calls `transport.close()`
without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts`
`#closeOnce`, an unresponsive adopted Runner that is not retained gets
`SIGKILL` after the close deadline. The Codex driver serves remote
Codex, OpenCode and ACPX runs, which matches the failures seen on
staging for all three driver kinds.

## What Changed

- `codex-app-server-driver-impl.ts`: `recoverSession` classifies a
thrown recovery failure. A transient one (`provider_transport_failed`, a
`PRP command ... timed out`, or the transport closing while waiting) now
detaches this controller with the transport's restart detach
(`detachControllerForRestart`). That path queues no `turn.stop` and no
`runner.suspend` and sends no signal, the same as a server restart. The
Runner and its in-flight turn stay for the same run's next reattachment.
An abort (a Stop or the recovery deadline), any message that names a
mismatch, a different session, identity, authority, integrity or
incompatibility, any unclassified failure, any returned identity
refusal, and a protocol integrity fault still close and stop the Runner.
`openSession` is unchanged.
- `codex-app-server-driver.recovery.test.ts`: the staging message
(`codex_history_read_failed: ... provider_transport_failed: PRP command
session.snapshot timed out`), and a plain snapshot timeout now detach
and never close. An abort of a hung read still closes and stops the
Runner. A thrown different-session refusal,
`native_adopted_provider_identity_mismatch`, an identity refusal that
also mentions a timeout, and an unclassified failure still close without
retention. The existing integrity-fault cases check that the close does
not retain the Runner.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of
69 (the two detach cases; master calls `close()` and stops the Runner).
- Green with the fix: 69 of 69.
- `npx vitest run src/drivers/codex src/native-session-runtime.test.ts
src/backends src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4` passed except two timing tests (Codex command
resolution, background cleanup containment) that pass alone and do not
touch this code.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- `detachControllerForRestart` is the existing restart path. It stops
the pump, disconnects the Runner, and drains the controller journal. It
queues no command and never signals the process.
- Not run: a live Daytona restart. The next post-upgrade audit should
show fewer `runner_remote_process_identity_unavailable` failures after a
`session.snapshot timed out` resume. Instead, the run log should show
"controller authority detached for restart; durable runner left alive"
and then a successful same-run reattachment.

## Risks

- Medium. A Runner that the recovery could not reach now stays alive,
with its turn, for the same-run reattachment. This is the state a server
restart already leaves. The same-run reattachment adopts it, or a Stop
cleanup or saved-work recovery stops it.
- If the Runner stays unresponsive across every same-run attempt, the
run no longer gets the `process_not_running` proof that #21039 uses for
its automatic fresh-session continuation. The run then keeps its visible
Retry. The staging data shows that reattachments reaching the Runner
sooner succeed, so a later attempt is expected to succeed. This is not
proven against a live sandbox.
- An abort of the recovery (a Stop or the recovery deadline) is
unchanged: it closes and stops the Runner.
`plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a
Stop during an adopted resume.
- No turn is replayed and no second Runner is launched. Adopting an
existing Runner never launches a new one.
- Previous-release story: no persisted state changes. The decision is
made in the process that runs the recovery, so a deploy during a run
changes nothing.
- No wait is added before a normal turn start. Only the recovery failure
path changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:59:46 -06:00
DottaandClaude Opus 5.5 b2cb8f8f56 test(native): cover a turn.start timeout that loses the remote Runner, end to end (#21041)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the same run
reattaches to that Runner
> - On staging, 31 tasks that a person retried are blocked. Their last
run failed with `runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`
> - In each run, the `turn.start` PRP command timed out. The close then
killed the Runner. The same-run reattachment found the Runner gone
(`process_not_running`, `providerTurnIssued: false`), and the task
waited for a Retry
> - #21023 and #21039 fix this sequence. Neither one is deployed yet. No
test reproduces this exact sequence, so a later change could break it
without a signal
> - This pull request adds two tests. One reproduces the turn start
timeout with a real `paperclip-runnerd`. The other reproduces the
staging failure record of a person's Retry
> - The benefit is that a regression of this failure class fails CI
before it reaches staging

## Linked Issues or Issue Description

Refs #21023, #21039, #20923, #20853

**What happened?**
A person retried a blocked task that runs Codex on Daytona. The Retry
started a fresh provider session and submitted a turn. The `turn.start`
PRP command timed out (`provider_transport_failed: PRP command
turn.start timed out`). The run closed. The run log shows `runner did
not prove durable suspension before checkpoint`, `runner shutdown
failed: Error: Durable recovery runner timed out.`, and then
`runner_remote_process_exited lifecycle=ready`. The log does not show
`remote runner left alive unsettled for same-run reattachment`. The
same-run retry tried to reattach 50 s later. It recorded
`native.remote_runner_recovery_failed` with `stage:
process_verification`, `reason: process_not_running`, `runnerSettled:
false`, `providerTurnIssued: false`, and an empty Runner stderr. The run
failed with `runner_remote_process_identity_unavailable`, and the task
waited for a person.

**Expected behavior**
A turn start timeout does not kill the Runner, so the same-run
reattachment adopts it. If the Runner is gone anyway, the run continues
once by itself in a fresh provider session and does not replay the turn.

**Steps to reproduce**
1. Start a remote native Codex run on a release before #21023.
2. Make the provider not answer `turn/start`, so the `turn.start` PRP
command times out.
3. The runtime cancels the session as failure cleanup. The close treats
that as a Stop and kills `paperclip-runnerd`.
4. The same-run reattachment finds `process_not_running`. Before #21039,
the completion hook refuses an automatic Retry with
`provider_turn_unsettled`.

**Paperclip version or commit**
Staging runs 8cefbe9977. That build has neither #21023 nor #21039. This
pull request is based on master after both.

**Deployment mode**
Hosted staging cloud workspace. Native runtime with a remote Daytona
sandbox.

**Agent adapter(s) involved**
Codex (native `codex_app_server` driver through `paperclip-runnerd`).

## What Changed

-
`packages/paperclip-runner/src/live/runnerd-turn-start-timeout-retention.test.ts`
(new): runs `executeNativeSession` over a real `paperclip-runnerd` with
a launcher-owned (remote-shaped) process. The fake Codex never answers
`turn/start`, so the `turn.start` PRP command times out. The test checks
that the run fails with that exact error, that the Runner process is
still alive after the close, and that its journal is `ready` for the
same-run reattachment.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`: a new
case with the staging failure record. A person's Retry (`task_retry`, a
user intent with a non-hook key, `forceFreshSession`) has only
`turn.submitted`. Its refusal has `runnerSettled: false` and
`providerTurnIssued: false`. The completion hook requests one automatic
Retry. The test checks one unknown-outcome event with `replayed: false`,
an idempotent second call, saved-work recovery first, and one queued
fresh-session successor. The fixture gets small optional inputs for the
wake, the events, and the refusal payload. Other cases are unchanged.
- No production code changes.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts`: 6 of 6 pass.
The new test passed 4 runs in a row.
- Red without #21023: with `native-session-runtime.ts`,
`harness-driver-backend.ts`, and `native-session-backend.ts` from before
#21023, the new runner test fails with `expected 'gone' to be 'alive'`.
That is the staging failure.
- `cd server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts`: 9 of 9 pass.
- Red without #21039: with `restart-interrupted-turn.ts` from before
#21039, the new server case fails with `{ requested: false, reason:
"provider_turn_unsettled" }`.
- `npx tsc --noEmit` passes for `server` and
`packages/paperclip-runner`.
- `pnpm verify` exits 2 because the new runner test file is outside the
resolved test graphs. CI supplies that coverage.
- Staging evidence: read-only GET of the failed run's events and log.
- Not run: a live Daytona run. #21023 and #21039 are not deployed yet.

## Risks

- Low. Tests only. No production code changes.
- The runner test starts a real `paperclip-runnerd` and the fake Codex.
It kills both processes and removes its temporary directory in
`finally`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 01:08:18 -06:00
DottaandClaude Opus 5.5 f7d41ca5bf fix(opencode): save an answer once when the model restates it after the completion call (#21025)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode models reply in chat through the Runner's
OpenCode driver. The driver settles one final response, and Paperclip
saves it as the reply comment
> - #20999 made the driver keep the prose written before the
`paperclip_finish` call together with the message written after it. That
stopped the loss of an answer when the later message only said "Answered
above"
> - In chat-eval cohort 4 (OpenCode with DeepSeek v4.1 flash), the model
often wrote its whole answer, called `paperclip_finish`, and then wrote
the same answer again in other words. The driver saved both. 13 cases
had one saved reply that held the answer twice
> - This pull request saves only the later message when it restates the
earlier prose. A short pointer line still keeps the earlier answer
> - The benefit is that people, deliveries and the next turn read the
answer once, and #20999's protection stays in place

## Linked Issues or Issue Description

Refs #20999

**What happened?**
An OpenCode chat turn wrote its answer, called `paperclip_finish`, and
then wrote the same answer again with small wording changes. The saved
reply comment held both copies, one after the other. Example
(budgets-costs.set-budget): "I can't set this from chat — changing an
agent's monthly budget is a board-only action ..." followed by "I can't
change agent budgets from chat — that's board-only ...", both with the
same two budget links.

**Expected behavior**
The saved reply holds the answer once. When the later message only
refers back to the earlier answer, the saved reply still keeps the
earlier answer.

**Steps to reproduce**
1. Use an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`.
2. Ask a question that the model answers before `paperclip_finish` and
again after it.
3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds
the answer twice.

**Paperclip version or commit**
master at f1aef0cb58.

## What Changed

- New `restatesEarlierReply(earlier, later)` in
`packages/paperclip-runner/src/drivers/opencode/restated-reply.ts`. The
later message restates the earlier prose only when all of these are
true: it keeps every link target and fenced code block of the earlier
prose, it is at least half as long, and it repeats at least half of the
earlier prose's distinct words. Identical text is always a restatement.
- `OpenCodeServerDriver` uses it in the path that #20999 added. When the
post-completion message restates the pre-completion prose, the driver
saves the post-completion message alone. Otherwise it joins both, as
before.
- The fake OpenCode server gets an
`answer-before-finish-then-restatement` turn that uses the recorded
cohort-4 texts.
- Tests: one driver test for the recorded shape, and unit tests for the
helper (identical, reworded, pointer line, dropped link, dropped code
block, different content).

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/restated-reply.test.ts
src/drivers/opencode/opencode-server-driver.test.ts`: 87 tests pass.
This includes the #20999 tests ("Answered above" keeps the earlier
answer; long commentary before a short post-tool answer keeps both).
- Red/green: with the driver file from `origin/master`, the new driver
test fails (the saved text holds both copies). With the change, it
passes.
- `npx tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.
- I ran the helper over all 18 recorded cohort-4 turns that had prose
both before and after `paperclip_finish`. It classified the 13 reported
duplicate replies (and 4 turns whose earlier line was a short status
that the later answer covers) as restatements. It kept both texts for
`access.cli-authorization`, where the later message is "Commands above
are ready to paste."
- A paid chat-eval run on a branch with this change is linked in a
comment below.

## Risks

- The check reads text, so it can be wrong at the margin. If it wrongly
calls an earlier answer a restatement, the saved reply loses that
earlier text. The link, code-block and length conditions keep the
references and the content of a substantial earlier answer; the earlier
prose remains in the run transcript.
- No saved run state, session state or event shape changes. A run saved
by the previous release resumes without change; only the text of a newly
settled final response differs.
- During a live run, the chat can still show the earlier prose as
progress above the saved answer, as it did before #20999.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:54:27 -06:00
DottaandClaude Opus 5.5 51d77648bc fix(native): keep a remote Runner alive when a failed turn start or event stream closes the run (#21023)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 stopped a failed close from force-stopping an unreachable
remote Runner. It asks for retention only when the session was not
cancelled
> - The runtime cancels the session itself when a turn start or the
event stream fails ("Native session event consumption failed."). That
cleanup sets the same flag as a user Stop
> - So after a failed turn start, the close still sent `SIGKILL` to the
Runner, and the same-run reattachment failed with
`runner_remote_process_identity_unavailable`. The task waited for Retry
> - This pull request separates failure cleanup from Stop. Failure
cleanup still revokes the turn's output, but the close keeps the Runner
for the same-run reattachment
> - The benefit is that these runs continue without Retry. Stop and
protocol integrity faults still stop the Runner

## Linked Issues or Issue Description

Refs #20923, #20853

**What happened?**
After the staging redeploy, five remote native runs failed with error
`runner_remote_process_identity_unavailable` and error code
`native_continuation_requires_reconciliation`. In four of them, the
first attempt started after the deploy. Its `turn/start` failed with
`runnerd did not report the provider turn identity`. The run log then
showed `runner did not prove durable suspension before checkpoint` and
`runner shutdown failed: Error: Durable recovery runner timed out.` The
diagnostic `remote runner left alive unsettled for same-run
reattachment` from #20923 did not appear. One second later the Runner
exited (`runner_remote_process_exited lifecycle=ready`). The same-run
reattachment 20–40 s later found `process_verification` /
`process_not_running` with `runnerSettled: false`.

**What did you expect to happen?**
The failed close keeps the Runner alive, as #20923 intended. The
same-run reattachment adopts it, and the run continues without Retry.

**Steps to reproduce**
1. Start a remote native Codex run.
2. Make `turn/start` fail (for example the Runner does not report the
provider turn identity in time), or let the event stream fail or time
out.
3. The runtime cancels the session as cleanup. `HarnessNativeSession`
treats that as a Stop, so `close` asks for `retainUnsettledRemoteRunner:
false`, and the transport kills the Runner.

**Root cause**
`packages/paperclip-runner/src/native-session-runtime.ts`
`attemptOptionalSessionCancellation` calls `session.cancel`.
`HarnessNativeSession.cancel`
(`packages/paperclip-runner/src/backends/harness-driver-backend.ts`)
sets `#explicitlyCancelled`, and `close` used
`retainUnsettledRemoteRunner: !this.#explicitlyCancelled`.

The fifth run is different. A server shutdown detached its controller
and left the Runner alive. Then the sandbox environment was acquired
fresh (45.9 s) before the reattachment, and the Runner was gone in the
middle of a turn. This pull request does not change that path. See
Risks.

## What Changed

- `NativeSession.cancel` accepts an optional `failureCleanup` field.
- `attemptOptionalSessionCancellation` passes `failureCleanup: true`,
except for a `NativeSessionProtocolIntegrityError`.
- `HarnessNativeSession` tracks a Stop separately from output
revocation. `cancel` with `failureCleanup` revokes the turn's output
only. `cancel` without it, and `revokeTurnPublication`, record a Stop.
`close` asks for retention when no Stop was recorded and no protocol
integrity fault occurred.
- Tests: three runtime cases with the real Codex backend (turn start
fails, event stream times out, Stop then failure) and a new backend
case.
- Docs: `doc/execution-semantics.md` describes the rule.

## Verification

- Red, on `master` with only the new tests: `cd
packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts -t "remote Runner retention after a
failed run"` fails 2 of 3 (`expected false to be true`): the close asked
to kill the Runner after the turn start failed and after the event
stream timed out. The Stop case passed.
- Green with the fix: the same command passes 3 of 3.
`src/backends/harness-driver-backend.test.ts -t "retain an unsettled"`
passes 3 of 3, including the failure-cleanup case. That case also checks
that the turn's result stays unpublished.
- `npx vitest run src/native-session-runtime.test.ts src/backends
src/drivers/codex
src/live/runnerd-codex-transport-remote-retention.test.ts
--maxWorkers=4`: 31 files, 814 tests pass.
`codex-protocol-integrity.test.ts` confirms that an integrity fault
still stops the Runner.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is
clean.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona run. The change is in the controller library
that runs on the server, so it needs no new Runner image.

## Risks

- Low to medium. After a failed turn start or event stream, an
unreachable remote Runner now stays alive instead of being killed. This
is the state #20923 already accepted for other failed closes. The
session stays quarantined, and the same-run reattachment or a Stop
cleanup owns the Runner.
- Stop is unchanged. A server Stop calls `cancel` without
`failureCleanup`, so the close still stops the Runner, also when the
stopped run later fails. A governed wait uses `revokeTurnPublication`,
which also records a Stop.
- A protocol integrity fault needs an operator and is not resumed on the
same run, so it still stops the Runner.
- Previous-release story: this change adds and changes no persisted
state. A Runner that an older release already killed keeps its current
named refusal and Retry. A deploy during a run changes nothing, because
the decision is made in the closing process.
- Not covered: a Runner that dies with its sandbox while the controller
is detached for a restart, in the middle of a turn. Its journal is a
settled `ready` cut with an active turn. #20853 resumes only idle
`ready` cuts. This needs a separate fix.
- No wait is added before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:21:05 -06:00
DottaandClaude Opus 5.5 50b79d711a fix(runner): keep call_api writes carrying their body (#21002)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agents use the `call_api` tool when no dedicated tool covers
an API operation
> - #20606 added an optional `saveResponse` flag to the shared
`call_api` schema for GET reads
> - After that change, OpenCode agents with DeepSeek v4.1 flash often
sent write calls without `body`
> - Body-less POST calls reached the routes and failed with 400; agents
then repeated the same call many times
> - This pull request moves the save option to its own read-only tool
and refuses body-less writes before they are sent
> - The benefit is that the write schema returns to its earlier shape,
and a missed `body` gets one clear, actionable error

## Linked Issues or Issue Description

Refs #20606

**What happened?**

In a full chat-eval cohort on master (OpenCode, DeepSeek v4.1 flash),
write calls through `call_api` arrived without `body`. The model's own
reasoning planned a body, but the emitted calls had only `operationId`,
`pathParams`, `query` and `saveResponse`. About 100 refused PATCH calls
listed exactly those arguments (the cohort before #20606 had none with
`saveResponse`). 72 POST calls reached the server with no body and got
400 (0 in the earlier cohort). One case sent the same body-less resolve
call 21 times.

**Expected behavior**

Write calls carry their request fields in `body`. When a body is
missing, the agent gets one clear error before anything is sent, and it
can correct the call.

**Steps to reproduce**

Run the OpenCode chat cases that resolve approvals, edit queued
messages, delegate with context, edit task properties or start preview
services on master after #20606. Count `call_api` writes without `body`
and POST 400 responses.

**Additional context**

The tool schema that OpenCode sends to the model is intact: `body` is
present. A short-context replay with the same schema did not reproduce
the omission (direct provider calls and local OpenCode runs: every write
carried `body`, with or without `saveResponse`). The omission appears in
long sessions, and it correlates with the added read-only flag. This
change removes that flag from the write schema and adds a safety net for
the remaining cases.

## What Changed

- Remove `saveResponse` from the advertised `call_api` schema and
description. The description now says that request fields go only in
`body`, and the `body` description says when it is required.
- Add a separate read-only `save_api_response` tool (`operationId`,
`pathParams`, `query`). It accepts only GET operation IDs, has no body
field, and uses the existing `saveResponse` export path with the same
access, sensitivity, budget and integrity rules.
- Refuse a write before dispatch when its body is missing: PATCH with a
required body (as before), and POST or PUT whose JSON request schema
names required fields. The error names the operation, the allowed
fields, the received arguments, the shape to resend, and that `query`,
`responseText` and `saveResponse` never carry request fields.
- Keep server-side acceptance of `saveResponse` on `call_api`, so saved
calls and receipts from earlier runs stay valid for recovery.
- Regenerate the runner contract, catalog, seeded fixture and manifest.
Update `doc/runner-api-tools.md`.

## Verification

- Red at master sources: the 3 replayed recorded writes (POST
resolve-from-comment, PATCH issue, POST comment, each with `query: {}`
and `saveResponse: false` and no body) were sent to the route instead of
being refused, and the schema test found no `save_api_response` tool. 4
tests failed.
- Green with this change:
`server/src/services/native-runtime/runner-api.test.ts` 2105/2105
passed; `runner-api.integration.test.ts` 61/61 passed;
`runner-bridge-tools.test.ts`, `task-document-key-contract.test.ts` and
`runner-api-artifacts.test.ts` passed; runner
`protocol-action-contracts.test.ts` 70/70 passed; server `tsc --noEmit`
passed.
- `runner-api-artifacts.integration.test.ts` now saves through
`save_api_response` and checks that a write operation ID or a body is
refused. Its download step needs Linux; it fails on macOS at master too,
so CI covers it.
- Paid E2E, OpenCode + DeepSeek v4.1 flash, [run
43138806990](https://github.com/paperclipai/paperclip/actions/runs/43138806990)
on this fix: 5/5 affected cases passed
(questions-and-approvals.reject-and-revise,
steering.edit-queued-message, task-coordination.delegate-with-context,
tasks.edit-properties, workspaces.preview-service). Body-less write
refusals went from 30 to 0, and body-less POST 400s went from 48 to 0,
compared with the earlier cohort on the same cases. The approval resolve
call was sent once (it was sent 24 times before). Every logged write
carried a body; the remaining non-2xx writes are the cases' intended
stale-edit and failing-fixture responses.
- Run-history redaction also covers `save_api_response` arguments; a
regression test fails on the earlier source and passes now.

## Risks

- Agents that learned `call_api` + `saveResponse: true` in a running
session will see a schema refusal and must use `save_api_response`.
Saved calls keep working for recovery.
- The new POST/PUT refusal applies only when the JSON request schema
names required fields, so writes whose body is optional are still sent.
- One more small read tool is advertised when API tools are enabled.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool
use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:46:14 -06:00
DottaandClaude Opus 5.5 5626a0299c fix(opencode): keep the answer written before the completion call in the saved reply (#20999)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode models (for example DeepSeek through
OpenRouter) reply in chat through the Runner's OpenCode driver
> - The driver picks one assistant text part as the turn's final
response. Paperclip saves that text as the reply comment and delivers it
to other channels
> - In a chat eval, the model wrote its full answer with links, called
`paperclip_finish`, and then wrote "Answered above — two inbox items
need your attention ..."
> - The driver saved only the later line. The chat showed both texts
from the run stream, but the saved reply, its deliveries and the next
turn's history held only a line that pointed at nothing
> - This pull request keeps all prose written after the last work tool
in the saved reply in that case, and lets the chat show it once
> - The benefit is that the saved reply holds the answer the person saw

## Linked Issues or Issue Description

**What happened?**
An OpenCode chat turn wrote its answer before the completion call, then
a short line after it that refers back to the answer. The saved reply
comment held only "Answered above — ...". The full answer with its links
existed only as progress text in the run stream.

**Expected behavior**
The saved reply holds the answer. Other channels, notifications and the
agent's next turn read the same answer that the person saw in chat.

**Steps to reproduce**
1. Use an OpenCode agent. Ask a question in chat that the agent answers
with links.
2. The model writes its answer, calls `paperclip_finish`, and then
writes a short line that refers to "above".
3. Read the saved reply comment with `GET /api/issues/{id}/comments`. It
holds only the short line.

**Paperclip version or commit**
master at 9dfdeaad86.

## What Changed

- `OpenCodeServerDriver` settles the final response from all eligible
prose parts when it selects a later message after the completion call
and the model also wrote prose after its last work tool and before that
call. It joins the parts in order with a blank line. Other selection
paths do not change.
- The chat transcript adapter removes the latest progress messages when
they compose the saved response exactly, so the text does not show
twice.
- Added a fake-server fixture and a driver test for the "answer, then a
line that refers back" turn. Updated the existing post-tool test: the
saved reply now keeps the earlier prose too, because structure cannot
tell commentary from an answer.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts` (80 tests pass).
- `cd ui && npx vitest run
src/components/task-chat/transcript-adapter.test.ts` (117 tests pass).
- Red/green: with the driver change removed, the new test and the
updated post-tool test fail. With the change, they pass.
- `npx tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes.

## Risks

- A turn that wrote commentary after its last work tool, then its answer
after the completion call, now saves both texts. The person already saw
both in chat. We accept this. Losing the answer is a worse failure than
keeping a short note.
- No saved run state, session state or event shape changes. A run that
the previous release saved resumes without change; only the text of a
newly settled final response differs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:19:38 -06:00
Nicky LeachandPaperclip 38501b1170 Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native agents open input images with a file reader.
> - Plugin Runner tasks use an authenticated Runner connection.
> - Runner needs a way to fetch an authorized file from the tenant app.
> - The tenant grants access to the exact attachment for the active run.
> - Runner downloads and verifies it before it returns a local path.

## Linked Issues or Issue Description

Refs #19740. Builds on the attachment authorization and command
transport added in #20939.

## What Changed

- Add the PRP v3 `runner.attachment.download` command for native
workspace providers.
- Issue short-lived grants for exact authorized wake and historical
attachments. Recheck the active run, source, size, and SHA-256 at the
tenant endpoint.
- Download current-wake inputs before provider dispatch. Download
historical files when the authorized chat reader asks for them. Preserve
SSH and sandbox command staging as a fallback.
- Run HTTPS transfers off the Runner control loop. Report completion
with one PRP event per transfer. Drop pending results on cancellation
and clear files at turn settlement, authority rotation, and restart.
- Grant restricted Codex sessions read-only access to the attachment
directory.

## Verification

- 69 targeted server attachment tests passed. They cover wake and
historical downloads, authorization failures, and command fallback.
- 22 focused historical attachment tests passed, including grant
redemption under a held database lock.
- Six focused Rust attachment tests passed, including a source-pinned
Codex sandbox check. The sandbox could read the staged file but could
not modify it or read adjacent Runner state.
- The real Runner transport test confirmed one durable command and one
completion event for a transfer. Restart-continuation tests confirmed
fresh attachment paths appear before dispatch.
- Server and Runner TypeScript checks, Rust formatting, and PRP
generated schema checks passed.
- Required GitHub CI passed on `7a33752bd7`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43122967837)
- Greptile scored the same revision 5/5 with no unresolved review
threads.
[Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134)

## Risks

- Saved provider and Runner identities keep their existing formats. A
previous Runner falls back to command staging where the target supports
it. A restart continuation refreshes temporary attachment paths before
dispatch.
- Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS
access to the configured tenant origin.
- Live agent-computer qualification remains a release check.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell tools, and
local test execution. Exact variant and context window are unavailable
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #123` /
`Closes #123` / `Refs #123` or (b) described the issue in-PR following
the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 16:28:02 -08:00
DottaandClaude Opus 5.5 1880526603 fix(runner): never start an OpenCode session without its Paperclip tools (#20964)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode agents use Paperclip tools through a local MCP
bridge in the runner
> - OpenCode 1.18 connects each MCP server one time per process. It does
not retry
> - One failed `initialize` or `tools/list` at startup marks the
Paperclip server "failed" for the full session
> - The agent then works with only built-in tools. In one chat it told
the person "Switched the working design to blue", but no saved state
supported this claim
> - This pull request makes the runner confirm the Paperclip tools while
the session loads, reconnect them, restart OpenCode if necessary, and
fail the session by name if the tools stay unavailable
> - The benefit is that an OpenCode turn never starts without its
Paperclip tools, so the agent cannot silently claim actions that it
could not do

## Linked Issues or Issue Description

Refs #20886, #20921.

**What happened**
In chat eval `steering.steer-active` (private paid run 43086418488, on a
commit with #20886 and without #20921), two of three OpenCode sessions
started without Paperclip tools. One was the CEO chat (fresh session).
One was the worker task after its saved session resumed in a new
OpenCode process. The CEO agent ran `opencode mcp list` and saw
`paperclip ✗ failed — Failed to get tools`. It then sent a request to
the bridge without the bearer token and got HTTP 401. That 401 came from
its own probe, not from OpenCode. The run still succeeded with the reply
"Switched the working design to blue …", because a plain chat reply
counts as the result.

**Expected behavior**
Every native OpenCode session starts with the Paperclip MCP tools. If
the tools cannot be loaded, the session fails with a named reason, and
the turn does not start without tools.

**Steps to reproduce**
1. Start OpenCode 1.18.34 with a remote MCP server whose first
`tools/list` request fails (for example, the server resets the socket
one time).
2. `GET /mcp` returns `{"paperclip":{"status":"failed","error":"Failed
to get tools"}}`. OpenCode keeps this status for the life of the
process.
3. `POST /mcp/paperclip/connect` returns `true`, and then `GET /mcp`
returns `connected` (about 8 ms). We measured this with the real 1.18.34
binary.

This is the same class as #20886: `Connection: close` made the one-shot
startup connect fail more often (about 9 of 25 sessions), and #20921
removed the trigger. Any other transient failure at startup still
produced a toolless session. This change removes the class.

## What Changed

- `opencode-server-driver.ts`: while a session opens (fresh, resumed or
recovered), the driver reads OpenCode's `GET /mcp`. If the `paperclip`
server is not `connected`, it calls `POST /mcp/paperclip/connect` up to
two times. If the server is still not connected, it throws
`paperclip_tools_unavailable`. The existing startup retry restarts
OpenCode for this error (three attempts in total). After that, the
session fails visibly.
- The check runs in parallel with session creation or resume. The added
work on a healthy start is one local HTTP request.
- `fake-opencode-server.mjs`: the fixture now serves `GET /mcp` and
`POST /mcp/paperclip/connect`. It connects to the real bridge with the
configured token, and it can simulate failed connects.
- New `paperclip-tools-startup.test.ts`: reconnect after one failed
connect, restart after a process stays failed, reconnect on a saved
session that resumes in a new process, and a named failure when the
tools never load. All four tests fail on master (master opens a toolless
session).

## Verification

- `pnpm exec vitest run
src/drivers/opencode/paperclip-tools-startup.test.ts` (in
`packages/paperclip-runner`): 4 passed. Red on master: 4 failed.
- All OpenCode driver, proxy, live runnerd OpenCode and native-runtime
server suites that use the fixture: 29 files, 567 tests passed.
- `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11
passed. `tsc --noEmit -p packages/paperclip-runner`: clean.
- Real OpenCode 1.18.34 probe (failure, then `connect`, then
`connected`) as in the steps above.
- Paid chat eval run 43117909094 on this change (steering.steer-active
and three other OpenCode chat cases). Result is in a comment on this PR.

## Risks

- Low. A healthy start adds one local `GET /mcp` request, in parallel
with session creation.
- If OpenCode cannot load the Paperclip tools after reconnects and
restarts, the session now fails with `paperclip_tools_unavailable`.
Before, it ran without tools. This is intended.
- No persisted run, session or pack state changes. A session saved by
the previous release resumes through the same open path and gets the
same check.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:07:38 -06:00
DottaandClaude Opus 5.5 d1e4109df1 fix(runner): keep a connected adopted Runner through a failed liveness probe (#20965)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, the controller adopts the remote Runner
that kept running in the sandbox. A monitor then checks the Runner's
liveness every 250 ms through the environment plugin.
> - #20940 and #20945 made a stalled plugin probe "unknown" for the
launched Runner's monitor, reattachment and launch. The adopted Runner's
monitor still failed the run on the first probe that failed without an
answer.
> - A short plugin stall after adoption therefore failed a healthy run
with `native_adopted_runner_identity_unverifiable`.
> - This pull request uses the evidence that is available: while the
adopted Runner holds its authenticated connection to the controller, a
liveness check that fails without an answer does not fail the run, for
at most the reconnect grace.
> - The benefit is that a healthy adopted Runner survives a short plugin
stall. A check that reports the process gone still fails the run at
once.

## Linked Issues or Issue Description

Refs #20940, #20945

**What happened?**
`#watchAdoptedRunner` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts` called
`failTransport("native_adopted_runner_identity_unverifiable: runner
liveness could not be revalidated")` on the first `isAlive()` rejection.
For a remote adopted Runner, `isAlive()` is a plugin
`environmentExecute` RPC. A plugin stall makes it reject, even while the
Runner holds its authenticated connection and works normally.

**Expected behavior**
A rejection is not an answer. While the Runner holds its authenticated
connection, the monitor keeps checking. The bound is the reconnect
grace. A check that returns `false` (the process is gone) still fails
the run at once.

**Steps to reproduce**
1. Adopt a remote Runner after a controller restart.
2. Stall the plugin worker, so that the liveness probe rejects.
3. Within 250 ms the run fails with
`native_adopted_runner_identity_unverifiable`.

**Paperclip version or commit**
master at ef755cecc6.

## What Changed

- `#watchAdoptedRunner`: on an `isAlive()` rejection, the monitor
records when verification was first lost. It does not fail the transport
while the control plane has an authenticated Runner connection and less
than `runnerReconnectGraceMs` has passed. A successful check resets the
clock.
- Unchanged: an `isAlive()` result of `false` fails the run at once. A
rejection with no Runner connection fails it as before. A transport with
no reconnect grace (local adoption) fails it as before. The close path
still uses `isAlive()` directly.
- `doc/execution-semantics.md` describes the rule.

## Verification

- Red on master: `npx vitest run
src/live/runnerd-codex-transport-adoption.test.ts -t "liveness probe
rejects while"` (in `packages/paperclip-runner/`). It failed with
`native_adopted_runner_identity_unverifiable: runner liveness could not
be revalidated`.
- Green: the test passes. The real adopted runnerd keeps its connection,
`isAlive()` rejects for 1.5 seconds (6 monitor ticks), the run still
answers `thread/read`, no signal is sent, and the runner is alive.
- `runnerd-codex-transport-adoption.test.ts` passes (24 tests),
including "retains authenticated adopted-runner ownership when a close
probe rejects". `tsc --noEmit` passes for `packages/paperclip-runner`.

## Risks

- A remote adopted Runner whose liveness cannot be verified, but which
stays connected, keeps running for up to 7 minutes before the run fails
closed. Its own authenticated connection is the evidence during that
time, and no exit is reported.
- The change touches `packages/paperclip-runner/src`, so it changes the
Daytona runner image content ID.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:49:13 -06:00
DottaandClaude Opus 5.5 d9d5958d09 fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path

- Paperclip's controller and the sandbox Runner exchange numbered
commands; the Runner refuses a command whose number is not exactly one
past the last it processed.
- Staging's most common "Runner gone" failure was a follow-up controller
whose journal was behind the Runner. #20928 removed the cause on the
server (a warm follow-up now launches from the journal its retired
predecessor saved) and added a defence that reconciles a journal behind
only by the predecessor's settled close.
- The Runner still treated an out-of-order command as fatal:
`durable/state.rs` returned an error and the process exited. A
controller with a stale journal from any other cause (state saved by an
older release, a mid-run deploy) therefore lost the Runner process and
the run could only fail as "Runner gone"
(`runner_remote_process_identity_unavailable`).
- This pull request makes the Runner refuse such a command with a
precise result instead of exiting, and makes the server rest that run
with a precise code and a Retry whose next run reconciles the journal
(the #20928 defence) before it opens the session.

## Linked Issues or Issue Description

- Refs #20928 (the server-side root-cause fix and the journal
reconciliation this relies on)
- Refs #20907 (makes runnerd exit text visible in the run log)
- Duplicate search: searched PRs for "controller sequence contiguous",
"controller_sequence_mismatch" and "out-of-order command"; none changes
the Runner's handling.

## What Changed

- `runner-core/src/durable/state.rs`: a new command whose
`controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap)
now yields `CommandDisposition::Reject` with `status: "rejected"` and `{
code: "controller_sequence_mismatch", message, expectedControllerSeq,
receivedControllerSeq }`. The command is not executed or journaled and
the last sequence is unchanged; the process keeps running. Replay of an
already-processed command and the compacted-history rejection are
unchanged.
- `native-session-executor.ts`: a failure carrying
`controller_sequence_mismatch` is classified as
`native_controller_journal_behind_runner`, a permanent (terminal)
failure with a Retry action that says no provider work began and that
Retry reconciles the journal from the saved checkpoint or stops with a
precise error.
- Tests:
- Rust `tests/durable_recovery.rs`: out-of-order (behind and gap)
commands are rejected with the precise code, the journal is unchanged,
and the next contiguous command executes. The existing
`command_gaps_and_identifier_reuse_fail_closed` unit test now asserts
the rejection instead of an error.
- Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a
follow-up controller with a journal missing the predecessor's close now
gets `"code":"controller_sequence_mismatch"`, the Runner process stays
alive and its durable sequence is unchanged; with the current journal it
resumes.
  - Server: the classification and terminal disposition of the new code.

## Verification

- Red (master Runner): the Rust test panics on `expect("an out-of-order
command is not fatal")`; the real-runnerd test receives `runnerd exited
unexpectedly with code 1: paperclip-runnerd: controller sequence must be
contiguous: expected 6, received 4`.
- Green: `cargo test -p paperclip-runner-core --test durable_recovery`
(4 passed); `cargo test -p paperclip-runner-core` lib suite (the only
other failure, `task_memory::tests::monitor_bounds_notices`, is a timing
test that passes alone); `npx vitest run
src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built
from this branch (2 passed); server `native-session-executor.test.ts`
focused tests (71 passed).
- `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`:
profiles match the release manifest. No ACPX profile, attestation, pin
or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the
change is Runner source behaviour with an existing command-result shape,
so no profile version or `commandDigest` changes.

## Risks

- A controller that is *ahead* of the Runner (a gap) is now rejected
instead of fatal. Neither case executes the command; the Runner journal
is unchanged, so this is strictly safer than exiting.
- The rested run needs a Retry. Before this change the same run failed
after its retries as "Runner gone" with no precise cause.
- Runner source changes the Daytona image content ID; a new image is
needed for chaos waves on this revision.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool
use (shell, file edits, tests) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:14:31 -06:00
DottaandClaude Opus 5.5 48d23d1530 fix(runner): keep a connected remote Runner through a stalled plugin supervision probe (#20940)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
monitors the process with a health probe that goes through the
environment plugin (`environmentExecute`).
> - A stall of the Daytona plugin worker makes that probe RPC time out
after 40 seconds. The monitor treated the timeout as "process
supervision failed".
> - The transport then failed closed with
`runner_process_termination_unconfirmed`. The run ended
`notification_transport_failed`, `recoverable: false`, and the close
path suspended a healthy Runner. The task rested blocked with Retry.
> - Nothing was wrong with the sandbox, the Runner or the provider. The
Runner held its own authenticated connection to the controller during
the full stall.
> - This pull request makes a failed or timed-out probe "unknown". While
the Runner stays connected on its own channel, the monitor keeps the
same runner and turn and probes again, for a bounded time.
> - The benefit is that a short plugin stall no longer kills a healthy
run. The fail-closed cases stay the same.

## Linked Issues or Issue Description

Refs #20909 (chaos cell CR3, which reproduces this failure)

**What happened?**
A 45-second stall of the Daytona plugin worker killed a healthy run.
Chaos cell CR3 reproduced it 2 of 2 times (OpenCode and Claude,
Daytona). A staging run had the same failure with no deploy: a plugin
`environmentExecute` RPC stalled for more than 40 seconds. The run
failed with "Notification transport failed closed:
runner_process_termination_unconfirmed: runnerd process supervision
failed: RPC call \"environmentExecute\" timed out after 40000ms". The
close path then sent `runner.suspend`, and the healthy Runner suspended
and exited. The task rested blocked with Retry.

**Expected behavior**
A probe that times out or fails in transport is not proof that the
process stopped. While the Runner stays connected on its own channel,
the run continues. If supervision cannot come back within a bound, the
run rests with a working Retry. A probe that positively reports the
process gone still starts the existing recovery.

**Steps to reproduce**
1. Start a remote (Daytona) run with a T1 workload.
2. Send SIGSTOP to the server's Daytona plugin worker for 45 seconds,
then SIGCONT.
3. The run fails `notification_transport_failed` about 40 seconds into
the stall.

**Paperclip version or commit**
master at 7cd5558e41.

## What Changed

- `createRemoteRunnerProcessLauncher`
(`server/src/services/native-runtime/native-session-executor.ts`): when
the health probe throws (RPC timeout or transport error) and the Runner
holds an authenticated connection, the monitor waits one poll interval
and probes again. It does not reject the process completion. The unknown
period has a bound: the remote runner reconnect grace (7 minutes). When
the Runner is not connected, the monitor keeps the old behavior: it
rejects after the 30-second probe budget, without exit proof.
- Probe results that positively answer (exit 45 "process gone", or
identity and generation failures) keep their exact current handling.
- `RunnerProcessLaunchSpec`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
has a new optional `isRunnerConnected` callback. `spawnRunner` passes it
to the process owner and keeps it on restart.
- The runnerd transport and the external runnerd driver supply
`isRunnerConnected` from the control plane's count of authenticated
Runner connections.
- `doc/execution-semantics.md` describes the unknown-probe rule and its
bound.

## Verification

- Red, on master: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "plugin
RPC timeouts until"` (in `server/`). The completion rejected after the
first stalled probe with `{ stage: "process_verification", reason:
"probe_transport_failed" }`. The transport turns that into
`runner_process_termination_unconfirmed`, which is the staging error.
- Red, on master: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t "live evidence of the
runner"` (in `packages/paperclip-runner/`). The launch spec had no
`isRunnerConnected` (`expected undefined to be true`).
- Green: both tests pass. New tests also prove:
- three 40-second plugin RPC timeouts with the Runner connected leave
the same runner supervised, and a later exit-45 probe still confirms
exit;
- with the Runner disconnected, the same timeout rejects without exit
proof (`probe_transport_failed`, `isExitConfirmed() === false`);
- with the Runner connected and probes that never answer, supervision
rejects without exit proof after the 7-minute bound.
- `server/src/services/native-runtime/native-session-executor.test.ts`,
`remote-runner-recovery-probe.test.ts`,
`packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts`,
`src/control-plane/durable-prp-control-plane.test.ts` and
`src/drivers/external` pass locally.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Chaos: CR3 results are in a follow-up comment on this PR.

## Risks

- While the Runner stays connected and the plugin cannot answer, the
controller waits up to 7 minutes before it fails closed. Before this
change it failed after about 40 seconds. During that time the Runner's
own connection is the liveness evidence, and the controller never
reports an exit.
- When the Runner is not connected, the timing is the same as before.
Sandbox loss still fails closed after the 30-second probe budget.
- The new spec field is optional. A process owner that does not read it
keeps the old behavior. State saved by an older version is not changed.
- No wait is added before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:45:25 -06:00
DottaandClaude Opus 5.5 884a9e19ea fix(native): launch a warm follow-up from its predecessor's saved journal, and restart a Runner that exited before session.open (#20928)
## Thinking Path

- Paperclip runs agent work through a controller (server) and a runner
(`paperclip-runnerd`) that exchange numbered commands; the runner
refuses any command whose number is not exactly one past the last it
processed.
- Staging's most common failure is "Runner gone"
(`runner_remote_process_identity_unavailable`,
`native.remote_runner_recovery_failed` with `process_not_running`,
`runnerLifecycle: ready`, `runnerSettled: false`). Chaos wave
`w-20261111T192658-s5201` (Daytona, warm, Codex, W2/W2Q/W2C re-wake
cells) captured it live: the new runner wrote `controller sequence must
be contiguous: expected 13, received 11` (and `expected 5, received 3`)
and exited 0.15 s after authenticating.
- In all three captured failures the follow-up run started 1–2 s before
the predecessor's warm session was closed (`transport_close_requested`
for the predecessor arrives after the follow-up's `run started`). The
close is issued by the follow-up's own admission.
- Admission first materializes the controller projection for the session
root. With an idle retained warm owner the root only holds the
external-persistence marker, so `restoreWarmRunnerdController` copies
that owner's saved checkpoint into the root. Later in the same
admission, `executePaperclipNativeSessionWithinScope` finds the owner
cannot be reused (configuration, process or GitHub access changed) and
retires it. Retirement sends `runner.drain` and `runner.suspend`; both
are saved to the owner's checkpoint before they are sent, but the root
still holds the projection taken before the close.
- The follow-up then opens its controller from that projection. Its
journal is two commands behind the runner's durable
`lastControllerCommandSeq`, so its first command repeats sequence 11;
the runner treats this as fatal and exits, and every later attempt only
reattaches to a process that is gone.
- So the saved controller journal was persisted correctly; it was not
selected: the projection is never refreshed after the retirement that
extends it.

## Linked Issues or Issue Description

- Refs #20874 (the runner-exit proof this extends from a rejected
`session.open` to a Runner that exited before answering it)
- Refs #20907 (makes the runnerd exit text, including `controller
sequence must be contiguous`, visible in the run log; independent of
this fix)
- Refs #20853 (related: resumes a remote Runner that died at a settled
`ready` cut; this PR covers the unsettled `ready` state left by an exit
before `session.open`)
- Duplicate search: searched open and closed PRs for "controller
sequence contiguous", "runner_remote_process_identity_unavailable" and
"Runner gone"; none fixes the stale controller projection.

## What Changed

- `server/src/services/native-runtime/warm-controller-journal.ts` (new):
- `refreshRetiredWarmOwnerProjection` reloads the retired owner's
canonical checkpoint into the session root after retirement, only when
the root still projects that owner and the saved journal extends the
projected one. A diverged journal holds admission with
`native_runner_checkpoint_projection_held:retired_owner_journal_diverged`;
a projection changed underneath holds with `...:journal_changed`.
- `reconcileControllerJournalWithRunner` is the defence for state
already saved in the bad shape (old-saved/new-resumed, mid-run deploys).
When the remote runner state shows its journal ahead of the controller
projection and the runner's last command is a completed
`runner.suspend`, it reconciles only if every extra sequence is a
completed `command_close_drain_*`/`command_close_suspend_*` in the
runner's processed-command journal and the predecessor's canonical
checkpoint holds exactly those commands. Anything else refuses before
launch with
`native_controller_recovery_missing_evidence:controller_journal_behind_runner`
instead of guessing. A runner whose last command is not a completed
suspend is left to the existing live-adoption checks.
- Defence in depth for the server side of the same failure
(`native_session_recovery_failed: runnerd exited while waiting for
session.open`, then `runner_remote_process_identity_unavailable` on
every same-run retry): a Runner proven to have exited while this
attempt's opening command (`session.open`, or the recovery probe) was
still unanswered is a failed fresh start, not a live Runner to reattach.
- `runnerd-codex-transport.ts`: `runnerExitConfirmed()` (#20874's proof)
now also holds for a resumed session whose provider identity was
preloaded, when the attempt's opening command is still pending at the
proven exit.
- The Codex driver's recovery result, `HarnessDriverBackend` and
`native-session-runtime` carry that proof on the
`native_session_recovery_failed` error.
- The executor records `runnerExitedBeforeSessionOpen` on the settled
failure; restart recovery puts `exitedBeforeSessionOpen` on the
`reattach_remote_runner` claim; `verifyRemoteRunnerOrSettledExit` and
`verifyStagedRemoteRunner` then start a new Runner when the process is
gone and the saved Runner state has no pending terminal delivery,
provider cleanup or warm transition (any lifecycle other than an active
turn). Attempt budget and identity checks are unchanged.
- `native-session-executor.ts`: calls the refresh right after the
configuration-changed retirement of a warm owner, and the reconciliation
in `restoreRemoteSuspendedRunnerCheckpoint` with the runner state it
already reads (no added remote read, no wait before `turn.start`).
- Tests:
- `native-session-executor.test.ts`: new `changed-external-checkpoint`
scenario of "rotates an owned warm codex remote runner before checking
changed configuration" with a real `DurablePrpControlPlane`: the retired
owner's close saves drain/suspend, and the journal the follow-up
launches from must end with them.
- `warm-controller-journal.test.ts`: refresh, divergence hold, untouched
marker/other run, reconcile of a settled close, refusal when the
canonical journal cannot prove the close, refusal for any other extra
command, and no action for a live runner.
-
`packages/paperclip-runner/src/live/runnerd-stale-controller-journal.test.ts`:
real `paperclip-runnerd` + fake Codex. A follow-up controller on the
predecessor's suspended runner with a journal that lacks the close
commands reproduces `controller sequence must be contiguous` and loses
the runner; with the current journal it resumes.

## Verification

- Red for the added part: `verifyRemoteRunnerOrSettledExit` with the
staging state (process gone, lifecycle `ready`, unsettled outbox)
rejects with `runner_remote_process_identity_unavailable` without the
change; with `exitedBeforeSessionOpen` it resolves to a fresh start, and
stays fail-closed for a pending cleanup, terminal delivery, active turn
or exhausted budget. Without the propagation, the backend recovery
result lacks `runnerExitConfirmed`; the real-runnerd stale-journal test
asserts the transport's exit proof after the contiguity exit (`expected
false to be true` on the old transport).

- Red on master (executor unchanged, new scenario): `AssertionError:
expected [] to deeply equal [ 'runner.drain', 'runner.suspend' ]` — the
follow-up launched from the projection taken before the predecessor's
close.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "rotates
an owned warm codex remote runner before checking changed
configuration"` (14 passed), `npx vitest run
src/services/native-runtime/warm-controller-journal.test.ts` (7 passed).
- Real runner: `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-stale-controller-journal.test.ts` (stale journal →
`controller sequence must be contiguous`; current journal → resumes).
- `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/paperclip-runner typecheck:typescript`; `pnpm verify`
selected broader verification (a new runner test file is outside the
resolved graphs), so broad coverage comes from required CI.

## Risks

- The refresh replaces the root projection only with a checkpoint whose
identity matches and whose journal extends the projection; otherwise it
holds admission with an existing
`native_runner_checkpoint_projection_held:*` code, which is safer than
launching a runner that will exit.
- The defence refuses launch when a runner is ahead by anything other
than a provable settled close. Before this change that launch always
ended in a runner exit and a failed run, so the refusal only changes the
error code and the moment of failure.
- This PR does not change the runner. The runner still exits on an
out-of-order new command; a follow-up runner PR will turn that into a
rejected command result.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool
use (shell, file edits, tests) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:41:03 -06:00
Nicky LeachandClaude Opus 5.5 d7b04a8b6d refactor(runner): rename codex-runtime-adapter to acpx-runtime-adapter (#20938)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner opens agent sessions through one ACPX runtime
adapter
> - The adapter began as the Codex ACPX adapter, so its file is
`codex-runtime-adapter.ts`
> - The adapter now opens sessions for every ACPX agent: Claude, Codex,
Pi, Cursor, Grok, Kimi, Copilot and Antigravity
> - The Codex name makes readers look for Claude and other agent
behavior in the wrong place
> - This pull request renames the module to `acpx-runtime-adapter.ts`
and changes no behavior
> - The benefit is a file name that tells readers what the module does

## Linked Issues or Issue Description

No issue exists. Refs #20933, which adds Claude restore behavior to this
module.

**What existing behavior does this improve?**
The name of the shared ACPX runtime adapter module in Paperclip Runner.

**Subsystem affected**
Paperclip Runner, ACPX driver
(`packages/paperclip-runner/src/drivers/acpx/`).

**Current behavior**
The module that opens ACPX sessions for all agents is named
`codex-runtime-adapter.ts`. Its test file, which holds Claude, Pi, Grok,
Cursor and Copilot tests, is named `codex-runtime-adapter.test.ts`.

**Proposed behavior**
The module is `acpx-runtime-adapter.ts` and its test file is
`acpx-runtime-adapter.test.ts`. All imports, scripts and docs use the
new name.

**Reason and benefit**
The name matches the module's scope. Reviewers and contributors find
agent-specific session behavior faster.

**Breaking changes**
None. No exported name, behavior or contract changes.

## What Changed

- Rename `src/drivers/acpx/codex-runtime-adapter.ts` to
`src/drivers/acpx/acpx-runtime-adapter.ts`, and its test file in the
same way. Git records both as renames with no content change, except the
test file's import path.
- Update the imports in `acpx-runtime-sidecar.ts`,
`codex-acpx-driver.ts` and four test files.
- Update the `dist` import in `scripts/grok-native-smoke.mjs` and the
test file key in `scripts/vitest-lane-durations.json`.
- Update the links and file names in
`doc/RUNNER-INTEGRATION-PLAYBOOK.md`,
`doc/architecture/runner-cursor-capabilities.md` and
`doc/architecture/runner-rich-acp-capabilities.md`.
- Do not change the historical profile identity fixtures under
`packages/paperclip-runner/test/fixtures/`. They record the sources of
earlier releases with the old path, and tests check them by hash.

## Verification

- `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`: no
errors in changed files. The only errors are in
`src/eval/workflow-harness.ts`, because
`@paperclipai/paperclip-eval-kernel` was not built in the local
worktree.
- `npx vitest run src/drivers/acpx
src/cli/acpx-runtime-sidecar.test.ts`: 93 files, 1859 passed, 9 skipped.
- `node --test scripts/contracts/contract-fingerprints.test.mjs`: 11
passed. The pack contract digest reads protocol schemas, the Rust crate
and the ACPX patch, not TypeScript sources, so the rename does not
change it.
- `node --check scripts/grok-native-smoke.mjs`, and
`scripts/vitest-lane-durations.json` parses.
- `git grep codex-runtime-adapter` finds the old name only in the
historical fixtures.
- Merge check with #20933: a local merge of that branch into this one
completed with no conflicts. Git applied #20933's changes to the renamed
files. The merged adapter tests passed (133 passed).

## Risks

- Low risk. The change renames files and updates references only.
- An open PR that edits the old file path must merge or rebase after
this PR. Git rename detection handled #20933 with no conflicts. A PR
that adds a new import of `./codex-runtime-adapter.js` fails typecheck
until it uses the new path.
- `openCodexAcpxRuntime` keeps its name. It is an alias of
`openQualifiedAcpxRuntime`, and this PR does not change exported names.
- `acpx-runtime-adapter.ts` and its test file were already longer than
1,000 lines. This PR does not add lines to them.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:29:25 +00:00
DottaandClaude Opus 5.5 c1edb9ec02 fix(native): keep an unreachable remote Runner alive when a failed run closes (#20923)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - On staging, `runner_remote_process_identity_unavailable` was the
most frequent failure. The tasks stayed blocked until someone clicked
Retry
> - In 31 of the 34 failed runs the reattachment found the Runner gone
with an unsettled `ready` journal (`runnerSettled: false`). The Runner
was not lost by accident. The controller's own close stopped it with
`SIGKILL`: close could not prove suspension within its 10 s budget, so
it force-stopped the Runner in the middle of a turn
> - The server then scheduled a same-run reattachment that could never
succeed, because the close had just removed the Runner it needed
> - This pull request keeps an unreachable remote Runner alive when a
failed run closes. A Stop, a cancelled turn and a `stop_cleanup`
recovery still stop it
> - The benefit is that the same-run reattachment finds the live Runner.
The Runner then settles into a `suspended` journal, which the existing
relaunch path resumes. The task continues without Retry, and no provider
work is replayed

## Linked Issues or Issue Description

Refs #20853, #20907

**What happened?**
Remote native runs failed with
`runner_remote_process_identity_unavailable` and error code
`native_continuation_requires_reconciliation`. The tasks were left
blocked. In the failed runs the first attempt failed with a recoverable
transport error, for example `provider_transport_failed: PRP command
turn.start timed out`, `native_session_interrupted` or a server restart.
The run log then showed `runner did not prove durable suspension before
checkpoint` and `runner shutdown failed: Error: Durable recovery runner
timed out.` The second string comes from `waitForProcess`, which sends
`SIGKILL` when its deadline passes. The same-run reattachment ran 20–60
s later and reported `process_verification` / `process_not_running` with
`runnerLifecycle: ready` and `runnerSettled: false`. Most of these runs
had done provider work before the failure (turns and tool calls), so the
automatic Retry correctly refused them (`provider_turn_unsettled`).

**What did you expect to happen?**
When a run fails for a reason that the server resumes on the same run,
the close does not destroy the Runner that the reattachment needs. The
reattachment adopts it, and the run continues without Retry and without
a repeated tool effect.

**Steps to reproduce**
1. Start a remote native run and let a turn start.
2. Make the Runner unreachable (a broken transport or a slow event
pump), so that the turn fails and the close cannot prove suspension
within its budget.
3. The close sends `SIGKILL` to the Runner. The same-run reattachment
then fails with `runner_remote_process_identity_unavailable`, and the
task is blocked.

**Relation to open PRs**
- #20853 covers the 3 runs whose journal was a settled `ready` cut
(`runnerSettled: true`). This PR does not change that path.
- #20907 makes the Runner's exit reason visible. It does not change
recovery. This PR does not overlap with it.

## What Changed

- `runnerd-codex-transport.ts`: `close(reason, {
retainUnsettledRemoteRunner })` accepts a new option. When the option is
set, the Runner is remote (a process launcher or an adopted Runner), the
close is not a `stop_cleanup`, the Runner is still alive and suspension
is not proven, the close does not stop the Runner. It skips the
`waitForProcess` kill, the adopted-runner `SIGKILL` and `forceKill`. It
still releases the controller route, writes the unsettled checkpoint and
throws `NativeSessionCloseUnrecoverableError`, so the session stays
quarantined and is never reused. The settlement diagnostic records
`runnerRetained: true`, and a new diagnostic records `remote runner left
alive unsettled for same-run reattachment`.
- `HarnessNativeSession.close` asks for retention only when the session
was not explicitly cancelled. After `cancel()` (Stop), it asks for the
old behavior, so Stop still stops the Runner.
- `CodexHarnessSession.close` passes the option to the transport.
`HarnessSession.close` and `CodexAppServerTransport.close` accept the
optional field. Callers that do not set it keep the old behavior.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`
describe the rule and the `runnerRetained` field.

How the run continues: the reattaching controller adopts the live
Runner. The Runner then completes the `runner.drain` and
`runner.suspend` that the failed close queued, and exits with a settled
`suspended` journal. That attempt fails as
`native_runner_process_exited`, which the server resumes on the same
run. The next reattachment finds the process gone with a settled
`suspended` journal and relaunches the Runner over it (the existing path
from #19745). If the Runner was only slow, it suspends by itself and the
first reattachment relaunches.

## Verification

- Red, on `master` with only the tests added:
`runnerd-codex-transport-remote-retention.test.ts` fails 2 of 5
(`expected 'gone' to be 'alive'`): the close killed the Runner.
`harness-driver-backend.test.ts -t "retain an unsettled"` fails 2 of 2.
- Green with the fix: `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/backends/harness-driver-backend.test.ts`.
- The staging shape: a real `paperclip-runnerd` started through a
process launcher, a turn in progress, the Runner frozen with `SIGSTOP`
during close. The close throws `NativeSessionCloseUnrecoverableError`
with `runnerRetained: true`, the Runner stays alive, and its journal
stays `ready`.
- A second controller attaches through `adoptExistingRunner` (no
duplicate launch). The Runner settles into `lifecycle: suspended` with
an empty outbox and no pending terminal delivery, then exits.
- An explicit cancellation, a caller that does not ask for retention,
and a `stop_cleanup` close still stop the Runner.
- Also run: `src/backends`, `src/drivers/codex`,
`src/drivers/acpx/codex-acpx-driver.test.ts`,
`src/drivers/acpx/codex-runtime-adapter.test.ts`,
`src/native-session-runtime.test.ts` (1045 tests pass; 2 `spawnSync
ETIMEDOUT` under full-parallel load pass alone), and all of `src/live/`
(files that timed out under full-parallel load pass at
`--maxWorkers=2`).
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is
clean.
- `pnpm verify` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. CI supplies that
coverage.
- Not run: a live Daytona run. The fix is in the controller library that
runs on the server, so it needs no new Runner image.

## Risks

- Medium. A failed close now leaves an unreachable remote Runner alive
instead of killing it. This is the same state a restart detach leaves
today, and the same-run reattachment or a Stop cleanup owns it.
- If the server does not resume the run on the same Runner, the Runner
stays alive until something reattaches or the sandbox is released.
Remote Runners have no reconnect deadline
(`runnerProcessReconnectGraceMs: null`). Before this change they were
killed. The close still quarantines the session, so it is never released
for reuse.
- Stop is unchanged: an explicit cancellation, `settleCancelledTurn` and
`stop_cleanup` close without retention, so they still stop the Runner.
- Previous-release story: this change adds and changes no persisted
state. State that an older release left (a killed Runner with an
unsettled `ready` journal) keeps its current handling: the named refusal
`runner_remote_process_identity_unavailable` and a working Retry. Tasks
already blocked on staging keep their Retry button. A deploy during a
run changes nothing, because the decision is made in the closing
process.
- No wait is added before `turn.start`. The change only affects the
failure path of close.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:17:25 -06:00
DottaandClaude Opus 5.5 64afd22d82 fix(runner): keep the OpenCode MCP bridge on keep-alive so OpenCode keeps its Paperclip tools (#20921)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode agents call Paperclip tools through a local MCP
bridge in the runner's OpenCode proxy
> - #20886 made the bridge send `Connection: close` on every response,
as hardening for one stalled parallel tool call that was never
reproduced
> - After #20886 merged, many native OpenCode runs started without any
Paperclip MCP tools
> - Agents then said the Paperclip tools were missing, worked only with
bash and file tools, and never called paperclip_finish
> - This pull request removes the `Connection: close` header and keeps
the delivery diagnostics from #20886
> - The benefit is that OpenCode agents get their Paperclip tools again,
and a recurrence of the original stall still shows in the run log

## Linked Issues or Issue Description

Refs #20886.

**What happened**
In chat eval runs on CI, every OpenCode session before #20886 sent a
first model request of about 34k–36k input tokens (prompt plus the
Paperclip tool schemas). After #20886, about 9 of 25 sessions sent only
about 8k–15k tokens on the same prompts (for example 14,711 for
questions-and-approvals.necessary-question, 15,481 for
cases.cases-disabled, 14,096 for cases.update-case-facts). The prompt
envelope size did not change (about 32.7k characters). Those sessions
made no Paperclip MCP calls: zero tool-delivery records, no
paperclip_finish. The agents' own reasoning said "The paperclip tools
aren't in my toolset". Sessions on a commit without #20886 (8d2e34eac1)
all had the full size.

**Expected behavior**
Every native OpenCode session starts with the Paperclip MCP tools.

**Steps to reproduce**
1. Run several chat-features OpenCode cells on master after #20886.
2. Compare the first usage record of each run (inputTokens plus
cacheReadTokens) and the tool names in the run events.
3. See sessions with about 15k tokens and no paperclip_* tool calls.

## What Changed

- `mcp-bridge.ts`: removed `response.setHeader("Connection", "close")`.
The bridge uses Node's default keep-alive again, as before #20886.
- The per-call delivery diagnostics from #20886 stay, so a recurrence of
the stalled parallel call still shows socket reuse, idle time and
`written` in the run log.
- `mcp-bridge-delivery.test.ts`: the pooled-client test now asserts that
the bridge does not close connections, that a later call reuses a
socket, and that each call still gets a correct delivery record.

## Verification

- `pnpm exec vitest run` for mcp-bridge-delivery, mcp-bridge,
mcp-bridge-diagnostics, opencode-proxy-finish-diagnostics,
opencode-app-server-proxy and opencode-proxy-abandoned-tool-call tests:
45 passed.
- The exact mechanism inside OpenCode/Bun is not reproduced locally (a
macOS Bun client against a `Connection: close` server completed 300
MCP-shaped handshakes without error). The evidence is the before/after
correlation above. The next chat eval runs will show the first-request
token counts again.

## Risks

- The original five-minute stall (one occurrence, cause not proven) can
recur. The diagnostics from #20886 will then name the socket state.
- No persisted run, session or pack state changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:10:56 -06:00
DottaandClaude Opus 5.5 951d1b0b1b fix(opencode): ask for the empty-response correction with reasoning off (#20890)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner drives OpenCode turns for agents that use OpenCode
models, such as DeepSeek through OpenRouter
> - Sometimes the model ends a turn inside its reasoning. It reasons
that it will reply and call `paperclip_finish`, and then it stops with
no answer and no tool call
> - The driver sends one correction prompt. The correction used the same
reasoning variant, and in the failed runs the model stopped inside its
reasoning again
> - The run then failed with `opencode_empty_response`, and the person
saw "The OpenCode run failed before it completed."
> - This pull request sends the correction prompt with the no-reasoning
variant, so the final answer comes in the answer channel
> - The benefit is fewer failed chat replies, with no extra prompts and
no change to the normal turn

## Linked Issues or Issue Description

**What happened**
A chat reply failed with `opencode_empty_response` in eval run
gha-43069861174 (`agent-chat.conference-room`, OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`). The first completion
and the one correction each held only a reasoning item (403 and 411
output tokens). Both reasoning items said the model would reply and call
`paperclip_finish`. No answer text and no tool call followed. In
retained eval evidence, 14 of 879 attempts needed this correction and 3
of those 14 failed. Each failed correction was another reasoning-only
completion.

**Expected behavior**
After an empty completion, the correction gets a final answer or a
completion report.

**Steps to reproduce**
1. Start an OpenCode turn with reasoning enabled on an OpenRouter model.
2. Let the provider return a completion that holds only reasoning.
3. See that the correction prompt uses `paperclip-default` (reasoning
on), so the model can end inside its reasoning again.

## What Changed

- `OpenCodeServerDriver` sends the empty-response correction with the
`paperclip-no-reasoning` variant. Normal turns keep their own reasoning
mode.
- `empty-response.test.ts` starts the `reply` and `empty-again` cases
with reasoning on. It checks that the first prompt uses
`paperclip-default` and the correction uses `paperclip-no-reasoning`.

## Verification

- `pnpm exec vitest run src/drivers/opencode/empty-response.test.ts
src/drivers/opencode/reasoning-mode.test.ts
src/drivers/opencode/interrupt-pending-tool-call.test.ts` (in
`packages/paperclip-runner`): 23 passed. Without the driver change, the
`reply` and `empty-again` cases fail (red/green).
- `node --test
tests/runner-e2e/native-completion-source-contract.test.mjs`: 29 passed.
- A paid eval run with this change and the related eval fixes is in
progress. The PR comments will have its result. One run cannot prove the
rate change, because the failure is rare.

## Risks

- Low risk. Only the correction prompt changes. A model with no
reasoning can give a shorter final answer, but the correction asks only
for the final answer and the completion report from work already done.
- Providers other than OpenRouter do not get a variant. Their behavior
does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:31:38 -06:00
DottaandClaude Opus 5.5 e601f9d2fc fix(runner): answer each OpenCode MCP tool call on its own connection (#20886)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip runner runs native OpenCode agents. OpenCode calls
Paperclip tools through a local MCP bridge in the OpenCode proxy.
OpenCode runs on Bun and uses Bun `fetch` for each MCP request.
> - In one chat eval, the model made two parallel tool calls. The bridge
produced and wrote both results in about 0.55 s. OpenCode received only
one of them.
> - The other call waited until Bun's default fetch timeout fired, about
five minutes later. The chat turn lost five minutes, and the eval missed
its deadline.
> - The most likely trigger is the reuse of an idle keep-alive socket.
We could not reproduce the loss locally, so this pull request is
hardening that is based on a hypothesis.
> - This pull request makes the bridge answer every request on its own
connection. It also records how each tool result left the bridge, so a
recurrence can be attributed from the run log.
> - The benefit is that a pooled socket can no longer strand a tool
result. If a call is stranded again, the run log shows whether the
bridge wrote the answer.

## Linked Issues or Issue Description

No public issue exists. Related: Refs #18751 (draft native client
acceptance diagnostic), Refs #19546 (abandoned queued calls), Refs
#20862 (transport event delivery; it does not touch this path).

**What happened**

In CI run `gha-42925998380` (chat-features, runner-opencode,
`cases.create-case`), the CEO chat run used OpenCode 1.18.34 with an
OpenRouter DeepSeek model. At 00:01:12.16 the model issued
`list_workflows` (MCP id 3) and `list_projects` (MCP id 4) in parallel.

The runner emitted the dynamic tool results for id 4 at 00:01:12.468 and
for id 3 at 00:01:12.714 (ok, HTTP 200). The bridge writes the HTTP
response directly after that event. There is no queue between them.

OpenCode completed `list_projects` at 00:01:13.113. It failed
`list_workflows` only at 00:06:01.761 with `The operation timed out.`

We read the bundled OpenCode source. Its MCP tool call has no timeout
that produces that message: the MCP request timeout gives `-32001
Request timed out`, and the configured value is 1 h for durable calls.
With the same binary in Bun mode (Bun 1.3.14), a fetch on a reused
keep-alive socket that never gets an answer fails after 298 s with
`TimeoutError: The operation timed out.` So Bun's default fetch timeout
ended the call, and the client never consumed the written answer.

The previous call (`list_agents`) answered at about 00:01:06.89, so its
socket was idle for about 5.27 s when the parallel pair started. Our
thesis is that call 3 reused that pooled socket, and call 4 opened a new
one.

**Expected behavior**

When the bridge has the result of a tool call, the MCP client gets it.
The client must not wait for a five-minute transport timeout.

**Steps to reproduce**

We did not reproduce it locally. This is the only occurrence in the
retained evidence. A macOS stress test (Bun 1.3.14 client, Node 26
server with the bridge's default keep-alive, one call, then an idle time
of 4.8–7.2 s, then two parallel calls, 8 pairs in parallel) completed
343 trials and about 1,029 requests with 0 lost results. The failing run
was on Linux CI with Node 24.

## What Changed

- `mcp-bridge.ts`: every bridge response sends `Connection: close`. The
MCP client cannot reuse a pooled keep-alive socket for a later request.
A loopback connect per request costs much less than the failure mode.
- `mcp-bridge.ts`: each `tools/call` records one closed delivery
diagnostic. This includes rejected calls (unsupported tool, invalid
input, duplicate identity), which record the outcome `error`
(`paperclip.opencode.tool-delivery-diagnostic/v1`) when its response
closes. The fields are the call id SHA-256, the socket request index,
the socket idle ms, the handler ms, the outcome (`result`, `error` or
`unanswered`) and `written` (`writableFinished`). It holds no tool
names, inputs or results. The limit is 256 records per bridge.
- `opencode-proxy-finish-diagnostics.ts`: the proxy forwards these
records to the run log as `warning` frames after session binding. It
validates the exact key set and value types and drops anything else.
- Tests: a new `mcp-bridge-delivery.test.ts` uses a pooling keep-alive
client with the incident's call shape (one call, then two parallel
calls). New forwarder cases cover the record validation. The existing
finish diagnostic assertion now filters by its own schema.

## Verification

- Red: with the `Connection: close` line removed,
`mcp-bridge-delivery.test.ts` fails with `expected 'keep-alive' to be
'close'`. The pooling client then reuses the socket.
- Green: `cd packages/paperclip-runner && pnpm exec vitest run
src/drivers/opencode/mcp-bridge-delivery.test.ts
src/drivers/opencode/mcp-bridge-diagnostics.test.ts
src/drivers/opencode/mcp-bridge.test.ts
src/cli/opencode-proxy-finish-diagnostics.test.ts
src/cli/opencode-app-server-proxy.test.ts
src/cli/opencode-proxy-abandoned-tool-call.test.ts` gives 6 files and 45
tests passed.
- The new test asserts that all three results arrive with the correct
JSON-RPC ids, that no socket is reused, and that each call records
`socketRequestIndex: 1`, `outcome: "result"`, `written: true` without
tool names or raw ids. A second test asserts that a client that leaves
before its answer records `outcome: "unanswered"`, `written: false`. A
third test asserts that a rejected call records `outcome: "error"`,
`written: true`.
- `tsc -p tsconfig.json --noEmit` for `packages/paperclip-runner`
reports no errors in the changed files. Seven existing errors in
`codex-runtime-adapter.ts` and `eval/workflow-harness.ts` come from this
worktree's linked dependencies. Required CI typecheck is the source of
truth.
- This does not prove the production fix. A recurrence on CI will show
socket reuse, idle time and `written` for the stranded call in the run
log.

## Risks

- Low risk. Each MCP request now opens a new loopback TCP connection.
That adds well under 1 ms per call, and there is no TLS.
- The `notifications/cancelled` handling and the abandoned-call abort do
not change. The abort still reads `writableFinished` on response close.
- One extra run-log warning for each tool call (at most 256 for each
bridge). The record holds only a hash, counters and timings.
- Previous-release story (upgrade-safe run state): no persisted run,
session, launch or pack state changes. A bridge from the previous
release keeps its behavior until the proxy restarts.
- If the thesis is wrong, the five-minute stall can recur. The new
records will then show `written: true` on a fresh socket and point the
next investigation at OpenCode or Bun.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:29:59 -06:00
DottaandClaude Fable 5.2 f9171ef621 fix(native): retry a throttled post-turn checkpoint for real; wait out storage throttling while saving the provider session (#20871)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native turn in a remote sandbox, closing the session saves
the provider session (the harness backup) to object storage. The next
attempt resumes from it.
> - On staging, object storage throttled that save ("Please reduce your
request rate", S3 `SlowDown`). The close failed, was retried 3 times 1 s
apart, and the session was quarantined. The next attempt of the same run
was refused with `native_session_cleanup_quarantined`.
> - The retries could never succeed: the Runner transport memoizes
`close()`, so every retry returned the first rejected promise and never
ran the checkpoint again. Quarantine recovery and admission recovery
call the same `close()`.
> - Also, the backup upload did not wait out a throttle or outage longer
than one request's own retries.
> - This pull request makes a close that failed only in its checkpoint
run that checkpoint again on the next `close()`, and makes the backup
upload wait out storage unavailability inside the existing bounded
storage wait.
> - The benefit is that a transient storage throttle no longer ends in a
quarantine that refuses the next attempt of the same run.

## Linked Issues or Issue Description

Refs #20772 (made the backup upload resendable after `SlowDown`). Refs
#20165 (per-request throttle backoff). Refs #20833 (earlier steps of the
same failure chain).

**What happened?**

1. A staging Codex run on Daytona finished its turn. The post-turn close
ran the remote checkpoint.
2. The checkpoint failed: `remote checkpoint failed: Please reduce your
request rate.`
3. The Runner retried the close 3 times 1 s apart (`native session
cleanup recovery after close failure (1..3)`), then 3 quarantine
recoveries. All failed within about 5 s.
4. The next attempt of the same run was refused:
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.

**Expected behavior**

A throttle slows the save down. A retry of the close saves the session
again. The next attempt of the same run proceeds once the save succeeds.

**Steps to reproduce**

1. Make a transport close whose checkpoint throws once.
2. Call `close()` again.
3. On master the second call returns the first failure without running
the checkpoint again.
4. Separately, answer each harness-backup `PUT` with 9 `503 SlowDown`
responses. On master `persistNativeHarnessBackup` throws `SlowDown:
Please reduce your request rate.`

**Paperclip version or commit**

master at 381b862135.

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: when
a close fails because its checkpoint failed (and the control-route
release did not fail), the transport keeps the checkpoint and the rest
of the close. The next `close()` runs the checkpoint again and then
finishes the close. All other close failures (for example a missing
suspension proof) stay memoized as before.
- `server/src/services/native-runtime/native-harness-backup-storage.ts`:
`persistNativeHarnessBackup` runs inside the existing storage outage
wait (`waitOutStorageOutages`, purpose "saving the provider session"). A
storage-unavailable failure (outage, throttle, rejected credentials)
that outlasts one request's retries is retried with the bounded
exponential storage wait (2 s doubling to 30 s, 5 minutes in total).
Each request still uses the jittered per-request throttle backoff from
#20165.
- Restart chaos: storage fault `ST8` answers the next 9 to 12 (seeded)
harness-backup `PUT`s with `503 SlowDown`. The storage audit reports it
when it never changed a request. The README documents it (for `--env
daytona --storage s3`).

## Verification

- RED on master (fix reverted, new tests kept):
- `runnerd-codex-transport-suspension.test.ts` "runs a failed close
checkpoint again on the next close": the second `close()` returns the
first `SlowDown` error.
- `native-harness-backup-throttle.integration.test.ts` "waits out a
throttle that outlasts one request's backoff": `SlowDown: Please reduce
your request rate.`
- GREEN: both pass. `native-harness-backup-throttle.integration.test.ts`
3 passed. `runnerd-codex-transport-suspension.test.ts` 12 passed.
`runnerd-codex-transport-release-resume.test.ts` 22 passed (twice).
`vitest --config tests/restart-chaos/vitest.config.ts`: 46 files, 554
tests passed, including the new ST8 planning and proxy test.
- `tsc --noEmit` clean for `packages/paperclip-runner`, `server` and
`tests/restart-chaos`.
- `runnerd-codex-transport-settlement.test.ts` has one case that also
fails on unchanged master in this worktree.

## Risks

- A checkpoint retry runs the same server checkpoint callback again
after the runner process was stopped. It reads the settled state from
the sandbox. If the sandbox is gone, it fails as before and the
quarantine stays.
- A close attempt can now take up to the 5-minute storage wait during a
long outage. This is on the post-turn path only, not before
`turn.start`.
- No change to the quarantine admission rules: admission still
accelerates one bounded recovery batch and refuses if it fails. Those
recoveries now really retry the checkpoint.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 12:11:16 -06:00
DottaandClaude Opus 5.5 a52f622be8 fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error

## Linked Issues or Issue Description

Refs #20808
Refs #20832
Refs #20820

**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.

**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.

**Paperclip version or commit**
master at 3ded7bbd91.

## What Changed

- Runner transport (`runnerd-codex-transport.ts`): new
`runnerExitConfirmed()`. It is true only after a completed close settled
the runner (provider stopped, runner suspended) and the runner process
exit was observed.
- Codex driver (`codex-app-server-driver-impl.ts`): after it closes the
transport for a failed session open, it sets `runnerExitConfirmed` on
the error when the transport proves the exit.
- Executor (`native-session-executor.ts`):
- A remote `provider_session_open_rejected` with a startup
`providerFailure` keeps the same-run `bootstrap_retry` only with that
proof. The saved failure records `runnerExitConfirmed: true`.
- Each failed attempt's recovery history entry records a startup failure
signature: provider, category and message, with numbers ignored. When
the next attempt fails with the same signature, it stops as
`native_provider_startup_rejected`. Its next action names the startup
error and keeps "Retry starts a new run".
- Deterministic rejections (`requires_action` authentication, permission
or invalid request) stop at once as before.
  - Local targets are unchanged.
- Restart recovery (`native-restart-recovery.ts`):
`remoteRunnerExitedBeforeProviderWork` also accepts a settled
`provider_session_open_rejected` with `runnerExitConfirmed: true`. All
its other checks still apply: no provider session, no provider event, no
checkpoint, no runner-sent event, no successor.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red on master, green here:
- `native-startup-rejected-retry.integration.test.ts`: proven remote
rejection → `retryable_failure` with `runnerExitConfirmed`; unproven →
terminal; repeat → terminal, the next action names `bwrap: loopback`,
and Retry is offered.
- `native-runner-restart-recovery.integration.test.ts`: proven failure →
`bootstrap_incomplete` (`remote_runner_exited_before_provider_work`); an
older release's failure without the proof stays held.
- `codex-app-server-driver.lifecycle.test.ts`: the driver records the
proof only when the transport reports it after close.
- `native-session-executor.test.ts`: unit cases for the rule, the
`cause` chain, the repeat signature and deterministic rejections.
- Local runs: the 3 server files above pass (1380 tests); restart
recovery `-t remote` 75 pass; driver lifecycle 25 pass.
- `tsc --noEmit` clean for `server` and `packages/paperclip-runner`.
- Not run: a Daytona chaos wave. There is no in-sandbox startup fault
seam yet (see Risks).

## Risks

- The proof depends on the transport's close settling the runner after a
failed `session.open`. The Rust runner keeps serving after a failed
`session.open` (lifecycle `Continue`) and accepts `runner.suspend`, so
close should settle. If it does not settle, there is no proof and the
run ends at once with Retry, which is today's behaviour. It is never
less safe than today.
- Retries stay bounded: the startup retry budget (3 attempts) and the
repeat rule both apply. A repeated identical error stops after one
automatic retry.
- Saved state: failures saved by older releases have no
`runnerExitConfirmed` and keep the old rule. History entries without a
signature never match a repeat. A mid-run deploy is safe: an old
executor never writes the proof.
- Not covered by chaos: a remote "Codex fails at startup N times" fault
needs a hook inside the sandbox Runner or Codex. DF1's seam is
server-side only, and no open chaos PR adds a seam inside the sandbox.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:55:15 -06:00
DottaandClaude Fable 5.2 76da2fec4f perf(runner): deliver transport events when they arrive instead of polling every 5 ms (#20862)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs each native agent session through a runner
transport. The transport reads the events that the runner commits to the
session's control plane and delivers them to the driver.
> - The transport found those events with a timer. Every live transport
ran a pass every 5 ms, for as long as the session was open, whether or
not an event had arrived.
> - A CPU profile of the staging server shows the cost. The main thread
was at 96% of one core, and this pass used 5.6% of it with about a dozen
live sessions. The cost grows with the number of sessions and never
stops.
> - The same timer also delayed every event. An event waited up to 5 ms
for the next pass before the driver saw it. Agent boot time and time to
first token are measured requirements.
> - This pull request makes the control plane tell the transport when a
reader can see a change, and the transport delivers in the next task.
The fixed timer is gone.
> - The benefit is that an open session with nothing to deliver costs
nothing on the main thread, and an event reaches the driver sooner than
before.

## Linked Issues or Issue Description

No public issue exists. The problem is described here.

**What happened?**

The server's main thread on a hosted instance stayed at 96% of one core,
so every request and every agent turn waited behind it. A 30 second CPU
profile of the live server (Node 24, about a dozen live native agent
sessions) attributes 5.6% of the main thread to `#pumpEventsSafely` and
4.7% self time to `#pumpEvents` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts`. Three
places started `setInterval(() => this.#pumpEventsSafely(), 5)`. Each
pass scans the session's committed-event window, so the cost is
`sessions x 200 passes per second x window size`, with nothing to
deliver.

**Expected behavior**

An open session that has no event to deliver does no periodic work. An
event that arrives is delivered at once, not at the next timer tick.

**Steps to reproduce**

1. `cd packages/paperclip-runner`
2. `PAPERCLIP_TRANSPORT_PUMP_BENCHMARK=1 pnpm exec vitest run
src/live/runnerd-transport-pump.benchmark.test.ts`
3. Read the `TRANSPORT_PUMP_BENCHMARK` lines.
`stateReadsPerSessionPerSecond` counts how often an open, idle transport
reads the control plane. `deliveryWaitMs` is the time from the moment a
reader can see a committed event to the moment the transport delivers
it.

**Paperclip version or commit**

Present on master at `c8ae02e5ce`. The profile is from `556b249897`.

**Deployment mode**

Hosted server with native agent sessions. The same code runs in local
development.

## What Changed

- `DurablePrpControlPlane` has a new `onReadableStateChange(listener)`.
It fires when the state that a reader of `store` can see may have
changed. The control plane calls it after every stored-state replacement
or save, after a saved checkpoint releases events that were waiting for
it, and after an authenticated runner moves the recovered cursor.
- The transport subscribes when it creates its control plane. A change
wakes one delivery pass in the next task (`setImmediate`). The pass
never runs inside the control plane's own commit, and several changes in
one task share one pass. This keeps the existing order: the control
plane finishes its commit before a reader reacts.
- The three `setInterval(..., 5)` timers are gone. `#startPump` and
`#stopPump` replace them and keep the same lifecycle: the pump runs from
session start, and it stops on release, controller retirement, failure,
and close.
- The delivery pass now reports what it left behind. Events that the
transport holds back itself (a pending turn start, a pending recovery
binding, a trace write that must be retried) are retried every 5 ms, as
before, and only while they are held. This is the only timer that
remains.
- A pass that stops after it exposes `turn/started` (so the driver can
bind the turn first) now continues in the next task. Before, the events
after it waited for the next 5 ms tick.
- `attachRun` wakes the pump when it moves the transport to the new run
identity.
- New regression test `src/live/runnerd-codex-transport-pump.test.ts`,
new opt-in benchmark
`src/live/runnerd-transport-pump.benchmark.test.ts`, a paced provider
fixture, and a test-only probe that records when each committed event
becomes readable.

How this meets `rules/sleep-aware-background-work.mdc`:

| Item | Answer |
| --- | --- |
| Trigger | The control plane's readable-state change. No timer. |
| Source of truth | The control plane's committed-event window and the
transport's delivery cursor. A notification is a hint to look at them. |
| Pending-work predicate | The pass returns `held` or `next_task`, or a
trace write is waiting. |
| Next deadline or retry | 5 ms, only while the predicate is true.
`next_task` uses one `setImmediate`. |
| Quiet condition | Nothing is held. Then the transport has no timer and
no scheduled task. Release, retirement, failure, and close stop it
completely. |

## Verification

**Reproduction and before/after measurement**

Run from `packages/paperclip-runner`:

```sh
PAPERCLIP_TRANSPORT_PUMP_BENCHMARK=1 pnpm exec vitest run src/live/runnerd-transport-pump.benchmark.test.ts
```

The benchmark opens 12 real transports against `runnerd` and a
credential-free provider fixture. It measures the open sessions while
nothing arrives. Then all 12 sessions stream one turn at the same time
(40 events each, one every 15 ms). "Before" is the unchanged source at
`e2cf9a670f`, which was the base of this branch when the measurements
were taken. "After" is this change on that base. Each cell shows two
runs, and the runs alternate between before and after. The host was a
shared macOS arm64 machine with other work running (load average 8 to
10), so absolute numbers carry that noise.

External checkpoint persistence. This is the mode that the server uses.

| Measure | Before | After |
| --- | --- | --- |
| Idle: control-plane state reads per session per second | 364 / 365 | 0
/ 0 |
| Idle: event-loop utilization, 12 new sessions | 1.5% / 1.5% | 0.0% /
0.0% |
| Idle: event-loop utilization, 49 retained events per session | 2.1% /
2.2% | 0.0% / 0.0% |
| Idle: process CPU, 49 retained events per session | 12.8 / 11.8 ms per
second | 0.05 / 0.04 ms per second |
| Delivery wait, mean (480 events) | 3.48 / 4.22 ms | 1.91 / 1.81 ms |
| Delivery wait, p50 | 3.16 / 3.97 ms | 0.83 / 0.58 ms |
| Delivery wait, p95 | 7.78 / 8.49 ms | 6.36 / 7.47 ms |
| Delivery wait, max | 8.84 / 9.23 ms | 9.07 / 8.86 ms |
| Delivery wait of the turn's terminal event, p50 | 2.37 / 2.58 ms |
0.08 / 0.05 ms |

The idle cost grows with the retained event window, as the profile
predicts. With `PAPERCLIP_TRANSPORT_PUMP_DELTAS=400` (409 retained
events per session, one run each): idle event-loop utilization 3.4%
before and 0.0% after, idle process CPU 21.5 ms per second before and
0.08 after. Delivery wait in that run: mean 5.93 ms before and 3.55 ms
after, p95 16.27 ms before and 8.03 ms after, max 66.94 ms before and
14.04 ms after.

Delivery wait is the time from the moment a reader can see a committed
event to the moment the transport hands it to the driver. It is equal or
better at every point. The p95 and the max did not move much. They
belong to the first events of each turn, which wait for the turn start
response. That wait is a separate 10 ms poll (`waitForRunnerCommand`),
and this pull request does not change it.

File persistence (the local journal, which writes and syncs on the main
thread):

| Measure | Before | After |
| --- | --- | --- |
| Idle: state reads per session per second | 363 / 364 | 0 / 0 |
| Delivery wait, 12 sessions, mean | 10.33 / 13.76 ms | 11.88 / 12.73 ms
|
| Delivery wait, 12 sessions, p50 | 6.04 / 6.87 ms | 6.34 / 6.20 ms |
| Delivery wait, 4 sessions and 400 events each, mean | 4.75 ms | 2.39
ms |
| Delivery wait, 4 sessions and 400 events each, p50 / p95 | 0.07 /
22.99 ms | 0.06 / 11.80 ms |

With 12 sessions the file journal saturates the main thread with its own
synchronous writes, and the wait is the time behind those writes. Three
more before-runs of that case gave means of 9.0, 14.0 and 20.2 ms, so
the two columns are the same within the noise. With 4 sessions the
journal does not saturate and the after column is better.

**Regression test (red, then green)**

```sh
pnpm exec vitest run src/live/runnerd-codex-transport-pump.test.ts
```

- `stops looking for events while an open session has none to deliver`
(file and external persistence). On the unchanged source: `expected 708
to be less than 100` and `expected 732 to be less than 100` state reads
in a 2 second idle window. With the fix: pass.
- `delivers each committed event in the next task, without a timer tick`
(file and external persistence). The test freezes every interval timer,
runs a full turn, and checks that each event is delivered before a task
that the test queues at the moment the event becomes readable. This is
an order check and does not depend on the speed of the host. On the
unchanged source: `the turn's events were delivered without a periodic
poll: expected false to be true`. With the fix: pass.
- Result: 4 of 4 failed on the unchanged source, 4 of 4 passed three
times with the fix, and 4 of 4 passed again after the rebase.

**Other local checks**

- `node ./node_modules/typescript/bin/tsc -p tsconfig.json --noEmit` and
`-p tsconfig.test.json --noEmit` in `packages/paperclip-runner`: pass on
the rebased commit.
- `pnpm exec vitest run src/live/runnerd-codex-transport` (15 files, 455
tests). These suites cover attachment, recovery, warm transition,
suspension, and release and resume on real `runnerd` processes. Before
the rebase, on a quieter host: all passed. On the rebased commit, at a
host load average of 15 to 20: 454 passed and 1 failed. The failure is
the terminal-replay row that is described below.
- `pnpm exec vitest run src/live src/control-plane
src/native-session-runtime.test.ts src/backends
src/drivers/dot/runnerd-dot-driver.test.ts
src/drivers/muse/runnerd-muse-driver.test.ts
src/drivers/codex/codex-protocol-integrity.test.ts --maxWorkers=3` (81
files): 1566 passed, 32 failed, 6 skipped. The failures:
- 30 failures in `runnerd-acpx-permission-ordering.test.ts` and
`runnerd-acpx-permission-forwarding.test.ts`. Both replace the control
plane with a fixture that appended events in place and relied on the 5
ms poll to notice them. The fixture now implements
`onReadableStateChange` and notifies its readers when it emits. Both
files pass (30 of 30).
- 1 failure in `runnerd-codex-transport.test.ts`, `stops a lost
runnerd's provider and escaped tool job from its lineage journal
(journal=true)`: `runner did not durably suspend before checkpoint`. It
passed 3 of 3 when run again alone.
- 1 failure in `runnerd-codex-transport-settlement.test.ts`, the
terminal-replay row of `settles only retained control authority without
starting another provider turn`: `native_cleanup_maintenance_unproven`,
with events still in the runner outbox. It failed again in 2 runs of
that test group and in the rebased run above, each time while the host
load average was 11 to 20. At a load average near 8 it passed 3 of 3
alone and 1 of 1 as a group with the fix. The unchanged master source
(`c8ae02e5ce`) gives the same result: it passed 2 of 2 alone and 1 of 1
as a group at a load average near 8, and the same row failed with the
same error (`native_cleanup_maintenance_unproven`, 27 events still in
the runner outbox) in a group run at a load average of 11 to 15. So this
row is sensitive to host load with or without this change. The failing
step is the retained-session maintenance path. That path builds its own
control plane, has no transport pump, has no listener on the new
notification, and has a fixed 30 second deadline.
- `pnpm verify` returns exit 2 (broader verification required, because
of the new fixture file). CI supplies the broad coverage.
- Not run locally: the server test suites, the repository-wide
typecheck, and the build. CI ran them on `4bce83f50a`: 59 checks passed
and 4 were skipped (Storybook deploy, Storybook visual regression,
Sandbox provider tests, Select Vitest files).

## Risks

- **A change that does not notify.** With no periodic pass, an event is
delivered only when something wakes the pump. Every wake comes from one
of a small, closed set: the five places where the store replaces or
saves its state, the checkpoint publication, the recovered-cursor
update, and the transport's own identity and start points. A new code
path that changes readable state without one of these would leave an
event undelivered until the next change. The regression test runs a full
turn with every interval timer frozen, in both persistence modes, so a
missing wake on the normal path fails the test. The existing transport
suites (attachment, recovery, warm transition, suspension, release and
resume) now run with no timer that could hide a missing wake.
- **Delivery in the next task, not later.** An event now reaches the
driver up to 5 ms earlier. The pass still runs in its own task after the
commit, and the same gates hold a terminal event until the turn start
response is released. The order of events does not change.
- **No protocol, schema, Rust runner, or provider pack change.** The
contract fingerprint inputs are untouched. Saved state from an older
release resumes the same way.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code. Tool
use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 11:08:50 -06:00
DottaandClaude Fable 5.2 381b862135 fix(native): let a finish through after one stale-result refusal; rebuild a missing instruction bundle (#20833)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run ends its task with a Paperclip terminal call
(`paperclip_finish` or `paperclip_block`)
> - The Runner refuses a terminal call that the agent decided before it
saw earlier tool results (`terminal_call_before_results`). This protects
against stale calls held across a restart.
> - On staging, Codex cancelled a nested exec cell. Codex never sent
`item/completed` for one Paperclip tool call in that cell. The Runner
then refused every later finish and block in the turn, and the turn
ended with no semantic result.
> - The same-run recovery then resumed the run on a new controller
container. The run's admitted instruction bundle is stored on the
controller's local disk, so it was missing there, and runnerd failed
with ENOENT.
> - This pull request makes a refusal name a returned call once, and
rebuilds a missing admitted instruction bundle before the Runner backend
starts.
> - The agent-directory lock part of the same failure chain ("Agent
directory operation is already active") is fixed separately by #20829.
> - The benefit is that the agent can finish after one refusal, and a
resumed run does not fail because a deploy moved it to a host without
the bundle.

## Linked Issues or Issue Description

Refs #20829 (the agent-directory lock wait and dead-owner release; this
PR does not change that lock).

**What happened?**

Staging Codex runs on Daytona sandboxes failed after about 30 minutes of
work:

1. Early in the turn, the agent called `paperclip_search_assigned_tools`
from a nested exec cell. Codex cancelled that cell. The controller
returned the result, but Codex never sent `item/completed`, so the
Runner never committed `tool.execution.completed` for it.
2. At the end, the agent called `paperclip_finish` four times and
`paperclip_block` once. The Runner refused each one with
`terminal_call_before_results` ("Those calls have now returned"). The
old call stayed "returned but unseen" for the rest of the turn.
3. The turn completed with no semantic result
(`native_finalization_missing`). Same-run recovery scheduled another
attempt.
4. On the new controller container, the attempt failed with
`runner_ingress_unavailable: runnerd process failed: ENOENT ... scandir
'/paperclip/instances/default/runtime-context-assets/bundles/<sha>'`
(one run) or `ENOENT ... lstat` of the same kind of path (a run during a
rolling deploy). In both, `<sha>` is the run's admitted instruction
bundle. The asset root is controller-local, and the new containers had
no volume.
5. The last attempt failed with "Agent directory operation is already
active" (fixed by #20829).

**Expected behavior**

The agent finishes after one refusal. A resumed run rebuilds its exact
admitted instruction bundle on the new host.

**Steps to reproduce**

1. Run a Codex task that calls a Paperclip tool, then
`paperclip_finish`.
2. Drop Codex's `item/completed` for the first tool call (new chaos
fault `--faults PV2 --provider-fault-message tool_completed`).
3. On master, every `paperclip_finish` is refused and the run ends with
`native_finalization_missing`.
4. Separately, delete `runtime-context-assets/bundles/<instruction
digest>` and resume a run that pinned it: runnerd fails with ENOENT.

**Paperclip version or commit**

master at 1d67e309e0.

## What Changed

- `packages/paperclip-runner/src/live/terminal-call-freshness.ts` and
`runnerd-codex-transport.ts`: when the Runner refuses a terminal call
with `terminal_call_before_results`, it records the refused call id with
the returned-but-unseen call ids it named. A later terminal call ignores
those calls only when the refused call's `tool.execution.completed` (the
refusal reached the provider) is committed before the later call starts.
A terminal call queued before the refusal reached the provider, a call
that was pending at the refusal, and a call made after it are judged as
before. The record is in memory, so after a Runner restart the agent can
get one more refusal.
- `server/src/services/native-runtime/runtime-context.ts`: new
`rehydratePinnedInstructionAsset`. When the admitted instruction bundle
is not present on this host, it rebuilds it from the agent's instruction
revision whose content has the admitted digest (single-file bundles), or
from the current instructions when they still produce the admitted
digest. It never substitutes different content. When nothing matches, it
leaves the context unchanged.
- `server/src/services/native-runtime/native-session-executor.ts`:
`createRunnerdBackendWithinSessionClaim` calls it before the backend
starts. When the bundle is present this is two local file reads.
- Restart chaos: new provider message `tool_completed` for the Codex
shim (PV1/PV2/PV3). It targets Codex's `item/completed` for the first
Paperclip tool call that is not a finish or block. Config accepts it
with A1 and C1 on Codex only. The README documents it.

## Verification

- RED on master (fix reverted, new tests kept): `vitest run
src/live/terminal-call-freshness.test.ts` in `packages/paperclip-runner`
fails with `expected [ 'paperclip_search_assigned_tools' ] to deeply
equal []`.
- GREEN: `terminal-call-freshness.test.ts` 13 passed; the transport
tests for terminal calls pass, including a new test where the first
finish is refused and the second is dispatched, and tests where a queued
stale terminal call and a late-returning pending write are still
refused.
- `server/src/__tests__/runtime-skill-preservation.test.ts`: 9 passed,
including the new test that deletes an admitted instruction bundle and
its manifest, then rebuilds the exact bundle from the revision history
while a newer revision exists.
- `vitest --config tests/restart-chaos/vitest.config.ts
provider-faults.test.ts`: 22 passed.
- Chaos wave `w-20261111T150212-s301` (`--env local --adapters codex
--workloads A1 --faults PV2 --provider-fault-message tool_completed
--tasks 1 --seed 301`): 1/1 PASS, `provider_fault_applied` applied=1,
R1–R6 and the A1 oracle pass.
- `pnpm verify` asks for broader verification (a restart-chaos README
change is a global CI input); required CI supplies it.

## Risks

- The terminal-call change lets a later finish through when a returned
call never reaches the agent. The earlier refusal named that call and
reached the provider first, so the agent decides with that knowledge.
- The instruction rebuild adds two local file reads before the Runner
backend starts. It writes only when the bundle is missing, and only
bytes whose digest equals the admitted digest.
- Not covered here: a multi-file (legacy) instruction bundle whose
content changed since admission still cannot be rebuilt; a skill bundle
with neither a version id nor a preservation receipt is not rebuilt; the
runner-side quarantine after an S3 `SlowDown` during the post-turn
harness backup.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 10:25:37 -06:00
DottaandClaude Opus 5.5 4529dfb77f fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only

## Linked Issues or Issue Description

Refs #20808

**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.

**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.

**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.

## What Changed

- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.

## Verification

- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in a3d43b7bfd reports
this case as 'sandbox failed to start' rather than a pass. Shell
commands in Daytona planning therefore do not run on the current image;
the driver doc records this as a remaining gap.
- Red/green unit test: `npx vitest run
src/drivers/codex/codex-security-config.test.ts` (in
`packages/paperclip-runner`). The new test failed on the unfixed code
(the profile was `":root"="none"` with `CODEX_HOME` denied and network
off) and passes with the fix.
- `cargo test --locked -p paperclip-runner-core --test codex_provider --
external_sandbox` and `cargo test --locked -p paperclip-runner-core
--lib -- external_sandbox_planning_profile_rules
runtime_launch_rebinding`: pass.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/codex-planning-read-only.test.ts
tests/runner-e2e/catalog.test.ts`: pass. `pnpm
test:e2e:runner:typecheck`: pass.
- Real Codex 0.160.0 app-server probe (macOS) with the generated launch
arguments and `PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`: `thread/start`
returned sandbox `{"type":"readOnly","networkAccess":true}` and loaded
the workspace `AGENTS.md`; `command/exec` with the planning profile
refused `printf x > plan-write-probe.txt` (exit 1, `Operation not
permitted`, no file) and allowed `cat AGENTS.md`. Trace logs showed 0
filesystem-helper invocations with the new profile and 336 with the
previous restricted profile.
- Codex 0.160.0 source: `exec-server/src/fs_sandbox.rs` always sets
`NetworkSandboxPolicy::Restricted` for the helper,
`core/src/agents_md.rs` uses the helper only without full-disk read, and
`linux-sandbox` adds `--unshare-net` only for a restricted or proxy-only
network.

## Risks

- Inside an external sandbox, planning commands can now read the whole
sandbox filesystem, including the isolated Codex home. Default-mode
turns in the same sandbox already have full read and write access, so
this does not widen what the sandbox exposes. Local and SSH planning are
unchanged.
- Inside an external sandbox, planning shell commands fail closed on the
current Daytona image because bubblewrap cannot map user IDs there.
Planning still reads files and writes the Plan through Codex and
Paperclip tools.
- If a future Codex release adds deny-read entries or makes the helper
mandatory, planning on Daytona could fail again. The new Daytona E2E
suite detects that.
- Upgrade safety: the profile filesystem and network keys are excluded
from the stable launch identity, so saved sessions resume with the new
rules (new Rust regression). No persisted schema, pack contract or ACPX
profile changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, GitHub CLI) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:44:28 -06:00
Nicky LeachandClaude Opus 5.5 ff8ec41e6a feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner and the shared `adapter-utils` engine run ACP
agents through the embedded ACPX library
> - Paperclip pins ACPX 0.13.1 and keeps a large patch
(`patches/acpx@0.13.1.patch`) on it
> - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`,
`processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly
launch) that make much of that patch unnecessary
> - A large patch is expensive to port and hides behavior that belongs
to Paperclip
> - This pull request upgrades to ACPX 0.22.0 and moves most patched
behavior into Runner and `adapter-utils`
> - The benefit is a patch that is 484 lines instead of 1,334, with the
moved behavior covered by Paperclip tests

## Linked Issues or Issue Description

Refs #20675, #14430, #15075, #17942, #9980

**Proposed enhancement**
Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the
patch behavior that stock 0.22 cannot express.

**Motivation**
The 0.13.1 patch held behavior that Paperclip owns, for example usage
receipts, Copilot policy, model checks and the agent environment. Each
ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let
Paperclip do this work itself.

**Current behavior**
Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1
code.

**Expected behavior**
Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only:
extension methods, Cursor subagent routing, event `_meta` and the
message boundary chunk, the protocol guard hook, `spawnAgent`, and
`requestExtension`.

## What Changed

- ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the
Daytona runner images and the qualified profiles (`acpxVersion`).
`@agentclientprotocol/sdk` moves to 1.9.0.
- New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1
patches are removed.
- Delivery receipts: Runner confirms a permission reply when
`onAcpMessage` sees the exact outbound reply. Question and extension
replies count as delivered when Runner hands them back to ACPX. New
`interaction-delivery.ts`.
- Copilot policy: the existing policy function runs from `onAcpMessage`.
A violation stops the agent, cancels later permission requests and fails
the next prompt through `assertActive`.
- Model selection: stock 0.22 validation, plus a sidecar gate
(`createAcpxEffectiveModelGate`) that refuses a prompt if the agent
reports another model. Grok binds its model when the session opens.
Antigravity adds its concrete model to the model list ACPX reads.
- Typed session failures: Runner and `adapter-utils` read them from the
turn result `_meta` and fail the turn. They advertise the capability
themselves.
- Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds
them to the turn's own `session/prompt` reply.
- Goal capability: Runner reads it from the `initialize` reply. It keeps
it under `paperclip.goalCapability` in the saved agent capabilities
`_meta` and restores it when a saved session loads.
- Agent environment: Runner and `adapter-utils` set the agent
environment in `spawnAgent`. ACP terminals keep their current
environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and
refuses to start if the server has an `ACPX_AUTH_*` variable.
- `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a
new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and
`processLifecycle`.
- Windows: ACPX 0.22 rejects raw command strings on Windows.
`adapter-utils` now passes an argv array there.
- Grok rules: a Runner guard (`grok-session-rules.ts`) writes
`session/new` `_meta.rules`.
- Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from
`onAcpMessage`. The adapter keeps a failed `dispatched` write and throws
it from the turn's `assertActive`. ACPX calls `assertActive` before the
prompt write, so the write still stops. The real-ACPX test now checks
this route. `docs/durable-recovery.md` is updated.
- Plan events: the sidecar reads 0.22 `status` events tagged `plan`.
- CI: the backup-checkpoint lane now installs through the shared
`setup-ci-dependencies` action, like PR CI, so it resolves a stale
`pnpm-lock.yaml` in place. It runs on any root `package.json` change,
and pull requests never commit the lockfile, so its own frozen install
failed on dependency changes.
- Tests that checked removed patch internals are removed or rewritten:
the response-delivery, Cursor-usage and Pi-receipt package contracts,
and the goal-persistence packaging test. New unit tests cover the moved
behavior.

## Verification

Local runs on macOS (darwin-arm64), Node 26, on the rebased head:

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx src/cli`: 2,000 passed.
- `tsc -p packages/paperclip-runner/tsconfig.json` and
`tsconfig.test.json`: no errors, except the existing
`@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`.
- Runner package contracts (`node --test
test/acpx-*-package-contract.test.mjs
test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`,
`scripts/contracts/contract-fingerprints.test.mjs`,
`scripts/acpx-patch-packaging.test.mjs`: all pass. `node
scripts/generate-acpx-sidecar-contract.mjs --check` passes.
- `cargo test -p paperclip-runner-core --test acpx_park_preparation
--test acpx_park_readiness --test acpx_warm_attachment --test
native_provider_backend`: 86 passed. `cargo fmt --check` passes.
- `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine
src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test
failed once under load and passes when run alone).
- Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and
39 passed), `grok-local` (149 passed).
- Server: `scripts/prepare-runner-vendor.test.mjs`,
`native-session-executor.test.ts`, `quarantined-task-retry.test.ts`
(1,416 passed).
- `cargo test -p paperclip-runner-core --lib -- previous_acpx_release
acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test
test/acpx-session-carry-forward-package-contract.test.mjs`: passes.
- `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output
identical to the port.
- Under full-suite load, some Runner live and recovery tests time out.
Each one passes when run alone. Full regression is left to CI.

Not run: live provider runs (they need provider credentials), the opt-in
Kimi live peer test, the Gemini filesystem smoke test, and Windows.

## Risks

- Previous-release story: sessions saved by the previous release (ACPX
0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote.
Runner moves a saved descriptor that names 0.13.1
(`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the
ACPX value in the session identity, so the saved session key and
identity record still match. The existing launch-profile rebind still
decides which sessions resume: a settled session, or the same run's
interrupted turn through lost-turn recovery. A session saved by any
other ACPX release is still refused as `provider_state_incompatible`.
`rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now
describe this.
- Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes
a session record written by the real ACPX 0.13.1 with its Paperclip
patch. 0.22 loads the same provider session (no `session/new`), runs a
turn and keeps the earlier turn. Rust tests cover a settled 0.13.1
session that the next run attaches to, and a turn interrupted by the
upgrade. A recovery-identity test checks that the session key and
identity record match across the versions.
- `pnpm-lock.yaml` is not committed. CI resolves it for each run, and
the lockfile refresh workflow updates it after merge.
- Behavior change on the server: `adapter-utils` refuses to create an
ACPX runtime if the server environment has an `ACPX_AUTH_*` variable.
Before, ACPX ignored such variables. Stock 0.22 would use them to
authenticate agents with server credentials.
- Delivery confirmation is now Runner-owned. A permission reply is
confirmed by the exact outbound message. Question and extension replies
are confirmed when Runner hands them to ACPX, not when the pipe accepts
them.
- Users can select only models that the agent advertises. Stock 0.22
rejects other model IDs (Claude is exempt).
- ACP terminals keep the sidecar or server environment, as before. This
existing exposure is not changed by this pull request.
- The `adapter-utils` Gemini version probe now runs with the server
environment plus the run variables, not the run variables alone.
- The pack contract digest and profile `acpxVersion` change, so Daytona
needs a new runner image.
- Grok launch flags (`--model`, `--reasoning-effort`) were checked on
1.0.13 but session open needs a login, so Grok keeps the existing
`_meta` path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, subagents). Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:28:00 -08:00
DottaandClaude fd1460bb24 fix(native): continue a governed-wait follow-up in a fresh session when its saved provider state is gone (#20809)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed

## Linked Issues or Issue Description

Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)

**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.

**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.

**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.

## What Changed

- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.

## Verification

- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live wave: a Daytona DF1 wave is running. Its result is added here
before merge.

## Risks

- Low risk for production. The hook reads its variable once at startup
and is ignored under `NODE_ENV=production`. Without the variable it does
one comparison and returns the same target object.
- The hook fails every sandbox command of an armed capture, including
the read-only fallback's inventory. A real full disk can let that
inventory succeed and fail later at the download. The outcome is the
same (`sandbox_disk_full`), but the fallback's inventory step is not
exercised under DF1.
- DF1 runs only with `--env daytona`. A local run has no sandbox
workspace save, so the harness refuses DF1 there.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 08:47:06 -06:00
DottaandClaude Opus 5.5 fa3cd6f330 fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.

## Linked Issues or Issue Description

**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.

**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.

**Paperclip version or commit**
master at 84fc67317e.

## What Changed

- Server (`native-session-executor.ts`): new permanent code
`native_provider_startup_rejected`.
- A definite `session.open` failure maps to it when its startup
`providerFailure` is `requires_action` for `authentication`,
`permission` or `invalid_request`.
- On a remote execution target (`nativeRemoteStartupRejectionCode`), any
definite `session.open` failure with a startup `providerFailure` maps to
it. A remote same-run bootstrap after the runner launched cannot prove
ownership and stops on `ambiguous_remote_recovery_evidence`.
- The run ends as `terminal_failure` with a board owner and a plain next
action. The board offers Retry, which starts one new run with a fresh
provider session.
- On a local target, unknown, transient, rate-limit, quota and model
startup failures keep the bounded same-run retries and end as
`native_session_retry_exhausted` with Retry. That code is unchanged
because the chat-channel failed-run retry depends on it.
- Runner: planning runs keep Codex's
`paperclip-runner-workspace-read-only` profile, also in an external
sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode
(Conversational)") and does not block writes or commands, so it cannot
replace that enforcement.
- Ask mode: an ask task builds `executionMode: "default"`. Its launch
has `readOnly: false`, the external sandbox profile, and no
collaboration mode. Ask mode restricts only Paperclip mutation tools,
not the Codex sandbox.
- Tests:
- `native-startup-rejected-retry.integration.test.ts` (new, embedded
Postgres, Codex provider): a deterministic rejection ends at once with
`canRetry: true`, and Retry starts exactly one new run. Unknown and
transient local failures stay `retryable_failure`. A spent retry budget
ends `terminal_failure` with `canRetry: true`.
  - Unit tests for the deterministic, local-retry and remote rules.
- `runner-provider-launch.test.ts`: ask, standard and planning launches
in an external sandbox.
- The Rust external-sandbox test now fails if a non-planning turn sends
a collaboration mode (`--forbid-plan-collaboration` switch on the fake
app server).
- Docs: `packages/paperclip-runner/docs/codex-driver.md`,
`packages/paperclip-runner/docs/durable-recovery.md`,
`doc/execution-semantics.md`.

## Verification

- Red on master, green here: the commands in "Steps to reproduce".
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1364 pass.
- `npx vitest run src/live/runner-provider-launch.test.ts`
(packages/paperclip-runner): 22 pass.
- `cargo test -p paperclip-runner-core --test codex_provider
external_sandbox`: pass. `cargo fmt --check`: clean.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Codex planning on Daytona still fails at thread creation. With this
change it ends at once with the provider message and Retry instead of an
ownership hold. A follow-up must make Codex's read-only sandbox work in
that container, or add a verified Paperclip read-only guard. This PR
does not relax read-only enforcement.
- A remote startup failure that could clear by itself now needs a Retry
click instead of an automatic same-run retry. Before this change, that
retry ended in an ownership hold, so no working automatic path is lost.
- A false `native_provider_startup_rejected` costs automatic retries and
shows Retry earlier. It never replays provider work, because
`session.open` precedes every prompt.
- Saved state: no persisted shape or launch argument changes. A run that
an older release held on `ambiguous_remote_recovery_evidence` after the
same rejection already offers Retry once its saved workspace is
recovered. No migration is needed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit, browser reads) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:48:18 -06:00
DottaandPaperclip 84fc67317e feat(runner): export authorized read responses as artifacts (#20606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native agents use authorized API reads to get task evidence.
> - Small responses return inline JSON, so an agent must copy each
document into a local helper.
> - Copying changes bytes and uses model output for data transport.
> - This pull request adds an explicit artifact export for non-sensitive
GET reads.
> - Helpers can download the exact response bytes and check their
identity.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The native `call_api` response capture path.

**Current behavior**

Small JSON responses have no artifact reference. Local helpers need the
model to copy the JSON data.

**Proposed behavior**

`saveResponse: true` saves a complete authorized GET response below or
above the inline threshold. Zero-byte success returns an explicit empty
receipt without storage. The receipt returns an artifact ID, byte size
and SHA-256 for `download_artifact`. Other callers keep the current
behavior.

Related work: #17393 covers saved app-tool result size. #18105 covers
mutation response paging. This change adds an explicit read export.

## What Changed

- Add GET-only `saveResponse` validation and capture through the current
artifact store.
- Refuse sensitive operations, denied reads, partial responses and
combined text paging.
- Keep route authorization, actor visibility, storage budgets and
download integrity checks.
- Update the native tool contract, generated catalogs and API guide.
- Add boundary and real-route byte comparison tests.
- Preserve saved run state. This is an optional input field; old calls
and receipts keep their current shape. No schema migration or identity
reset is needed. The seeded catalog fixture is generated.

## Verification

- Final published head `d47b52e1bf412418e24768f865cc139097a6d068`:
[typecheck, build, all tests, Linux Rust and aggregate verify
passed](https://github.com/paperclipai/paperclip/actions/runs/42918883477);
[Windows Rust
passed](https://github.com/paperclipai/paperclip/actions/runs/42918883348);
[quality gate
passed](https://github.com/paperclipai/paperclip/actions/runs/42918883016).
Greptile completed successfully at this exact head, 5/5; all five review
threads resolved. Automated Paperclip review remains queued due to a
failed reviewer run; no final verdict is claimed for that service.
- Current head: `d47b52e1bf412418e24768f865cc139097a6d068`. Merged
current master to refresh GitHub's stale base snapshot; the PR now
contains exactly nine intended artifact transport files, with no
lockfile or UI changes. Fresh final-head CI/review is required.
- Previous head `986ad87490fff93cfec313ba6fd56d3dc0bb665b` passed
typecheck, build, Rust verification, artifact/native suites and
completed test shards; remaining shards were still running when the base
snapshot was refreshed. Greptile was 5/5, with one unrelated master UI
finding resolved as outside the corrected PR diff.
- Review fixes verified at `986ad87490fff93cfec313ba6fd56d3dc0bb665b`:
2,135 checks passed across the five focused suites; Runner TypeScript
build and regenerated contract/catalog checks passed. This head replays
the original implementation on current master and retains the original
PR commit in its history.
- Review red/green at replayed baseline `da3a1d4de9`: empty HTTP 200/204
tests failed with real storage's `File is empty`, then passed with zero
new assets or quota reservations. Error-body tests failed without
size/truncation fields, then passed.
- Empty successes now return `empty: true`, `byteSize: 0`, `data: null`
without storage. Nonempty JSON results such as `[]` export normally.
Error receipts report byte size and truncation. Docs and tool guidance
state the existing 10 MiB download limit and paged text alternative.
- Previous-head CI typecheck/build/tests passed except the Rust
process-lineage cleanup test (`unattributed_orphan`); final-head CI and
fresh review remain required.
- Red at `7368a0b15c`: the small-document export test failed with
`Invalid call_api arguments` before production edits. The same check
passed after the change.
- Final post-rebase run at `6e7a35ed6a59bb74f4218312d9f10557099433c3`:
all 2,124 checks passed across the API, capture, artifact boundary,
real-route export and artifact download suites.
- Real-route export test passed: HTTP bytes, stored bytes and downloaded
bytes were equal. Foreign-company access and forged download metadata
were rejected.
- Existing API integration and artifact suites passed. Nearby API
integration checks also passed (59 passed, 2 skipped).
- Disposable native helper scan passed: 12 existing documents, 15 GET
requests, zero mutations, 73,919 bytes, 17.985 seconds. All files passed
size/hash checks. The bridge passed file references only. Production
token savings were not measured.
- Runner TypeScript build and generated contract checks passed.
- `pnpm verify` returned exit 2 and required broader CI coverage. Local
preflight stopped at missing Cargo after dependency and SDK setup. The
local server typecheck stopped during the cycle layer without a
diagnostic; the container recorded an OOM kill. This check has no pass
verdict. Required CI must cover typecheck, build and test at the final
head.

## Risks

- Each explicit export uses the current capture budget and storage
admission. Frequent small exports can reach that budget.
- An artifact holds one HTTP response. Callers must still follow list
pagination.
- No database or saved-state migration is required. The option does not
change default response behavior.

## Model Used

OpenAI `gpt-6.1-sol`, through Codex in Paperclip Runner. Used reasoning,
shell execution and native tools. The active agent configuration
supplied this exact model ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 06:26:41 -06:00