Commit Graph
567 Commits
Author SHA1 Message Date
Nicky LeachandClaude Opus 5.5 ff4ba7d319 perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and
proxies from qualified launch artifacts: a Node runtime and the scripts
it runs
> - Since #12699, Runner copies each artifact into a sealed memfd and
checks its SHA-256 on every verification. This defends against a path
swap or an in-place rewrite between the check and the exec
> - On a hosted agent computer (4 vCPU), the copy and digest of the 127
MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake.
This is on the path from `session.open` to the sidecar spawn, and again
on every warm `run.attach`. Each Runner also holds about 125 MiB of
shmem for the copy
> - On that computer, the artifacts live on the Runner ROM: a read-only
erofs image that the deploy pins by digest. The threat that the memfd
defends against does not exist there. On local hosts the artifacts are
writable, so the memfd must stay
> - This pull request adds an explicit, path-scoped, default-off
`--trusted-launch-root <dir>`. Runner runs an artifact in place only
when its open descriptor proves that it is on an immutable mount under
that root. Everything else keeps the sealed, hashed snapshot
> - The benefit is a shorter `session.open` and `run.attach` on agent
computers, and about 125 MiB less shmem per Runner, with no change where
the flag is absent

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Launch-artifact verification in `paperclip-runnerd`
(`qualified_launch.rs`). Its callers are the ACPX `verified_transport`
(every `session_config`, including the park rebind) and the OpenCode and
Cursor Cloud proxy launches.

**Subsystem affected**
Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`).

**Current behavior**
Every verification opens the artifact, copies all bytes into a sealed
memfd, and hashes them. Node is 127 MB. The cost is described in the
Thinking Path.

**Proposed behavior**
When the host passes `--trusted-launch-root /opt/paperclip/runner` and
the root passes its checks, Runner holds each admitted artifact open in
place and executes it through `/proc/self/fd/N`. It does not copy or
re-hash it. Without the flag, nothing changes.

**Reason and benefit**
See the Thinking Path.

Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see
below).

## What Changed

- `qualified_launch.rs`: new `TrustedLaunchRoot` and
`set_trusted_launch_root` (process-global, set once at startup, like
`--task-memory-cgroup`).
- **Root checks at startup.** The path must be absolute, already
canonical (no symlink or `..`), a directory, and on a read-only,
exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a
warning and keeps today's behavior.
- **Per-artifact checks, on the open descriptor.** After today's checks
(regular file, not a symlink, not group- or world-writable, open-time
identity), the descriptor must be a regular file on the root's device.
Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by
path components, so `/opt/paperclip/runner2` does not match. Its mount
must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs`
(erofs or squashfs magic). If any check fails, Runner logs the reason
and uses the sealed snapshot.
- **Admitted artifacts** become
`VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The
held file goes through the existing `inherited_artifact` path, so the
exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`
still names a descriptor that nested launches can reopen.
- New `verify_assigned_executable_launch_artifact` never uses the root.
`stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always
keep the sealed snapshot.
- `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older
runnerd builds ignore unknown flags, so a host can pass it before every
Runner understands it.
- The trusted path is Linux-only. On macOS and Windows,
`TrustedLaunchRoot::open` always refuses.
- `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi
provider-death fixtures behave under a trusted root.

### Decision: the trusted path drops the digest

The trusted path does not hash the artifact, not even once per process:
- The ROM image is pinned by digest at deploy time.
- On the agent computer, the Launcher computes the expected
`--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from
the same ROM files immediately before it starts Runner. A re-hash of an
immutable file can only repeat that result.
- One hash per process would not help `session.open`, because each task
starts a new Runner process.
- The mount check is the guard: erofs and squashfs have no write path,
so the bytes cannot change while the descriptor is open.

### Fixtures that assert the sealed memfd

- Untrusted-mode assertions are unchanged. The fixtures do not pass the
flag.
- Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a
trusted root, the parent descriptor is the ROM's Node file (one link).
The existing linked-descriptor branch handles it: same inode as
`/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`.
That branch never accepts the file as a memfd. The existing test
`test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim`
covers it.
- Pi provider-death admission binds `/proc/self/fd/3|7` to the native
distribution snapshot's Node. The TypeScript ACPX closure copies that
file whether or not a trusted root is set, so the flag does not affect
it.

### Overlap with #20423

#20423 (draft) moves the same copy and hash to Runner boot and reuses
the result. Where this flag is set (Cloud agent computers), there is
nothing left to precompute, so #20423 is mostly redundant there. #20423
still helps hosts without a trusted root (local and sandbox Runners).
This PR does not change #20423.

## Verification

All runs are on Linux (aarch64, Rust 1.97.1, in a container):
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`
- `cargo check --manifest-path runner/Cargo.toml --locked --workspace`
- `cargo test -p paperclip-runner-core --lib -- qualified_launch
trusted_root commonjs`: 12 passed. The new tests are:
-
`an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`:
trusted-root hit, with no memfd, the same inode as the source, and no
re-hash.
- `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a
path outside the root is sealed, and a wrong digest still fails.
- `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`:
`rom2/` does not match `rom`.
- `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the
real probe refuses a writable root at startup. A per-artifact mount
failure falls back, and a wrong digest still fails.
  - `the_real_mount_probe_rejects_a_writable_filesystem`
- `symlink_and_parent_escapes_from_the_root_fall_back`: a directory
symlink out of the root, a `..` path out of the root, and a
final-component symlink (still refused).
  - `a_root_must_be_absolute_canonical_and_a_directory`
- `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag
absent, behavior unchanged.
  - `host_assigned_executables_ignore_the_trusted_root`
-
`trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`:
Runner execs from the held file, and a child re-execs
`$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as
`/proc/self/exe`.
- `cargo test -p paperclip-runner-core --test stdio_mcp --test
process_supervisor`: 4 and 8 passed.
- Ad hoc, not committed: a privileged container with a real loop-mounted
squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and
refused a `ro,noexec` mount. The unit tests replace only that mount
probe, because CI cannot mount filesystems.
- Not verified: a run on an agent computer with the real erofs ROM, and
the latency and memory savings there. The Cloud Launcher change that
passes the flag is a separate follow-up.

## Risks

- **This change needs review from a security owner before merge.** It
removes the sealed copy and the per-launch digest for admitted
artifacts. That relies on the host's assertion that the root is a
digest-pinned immutable image, plus the descriptor checks above.
- **Misconfiguration fails closed.** A wrong, writable, `noexec`,
non-canonical or missing root only loses the speedup, and runnerd logs a
warning. There is no global "trust everything" switch, and Runner never
selects a root from the filesystem it finds.
- **Residual trust.** On an admitted file, Runner no longer detects a
ROM image that differs from what the launch profile expected. On the
agent computer, the Launcher derives the expected digests from the same
ROM. Another host that passes the flag must pin its image the same way.
An attacker with root in the guest (who could write the backing block
device or remount) is out of scope; such an attacker can already control
runnerd.
- **Allowlist.** Only erofs and squashfs are accepted, because they
cannot be written through any mount. A read-only bind mount of a
writable filesystem is refused. A ROM that changes to another filesystem
type will silently fall back to the sealed path (slower, still safe).
- No migration. Behavior is unchanged when the flag is absent. Older
runnerd builds ignore the flag.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, container builds). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:46:33 -08:00
DottaandClaude Opus 5.5 60a60a0cd1 fix(opencode): attribute unreadable tool-result images to the tool (#20399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner drives OpenCode agents. It sends their
OpenRouter and Paperclip-connection model calls through a local provider
proxy.
> - An agent often takes a screenshot of its own work and reads it to
verify the result.
> - OpenCode 1.18.34 cannot put images in OpenAI-compatible tool
results. It moves them into a synthetic user message ("Attached media
from tool result:"). For a text-only model, it replaces the image with
"ERROR: Cannot read image (this model does not support image input).
Inform the user."
> - The model reads that text as a message from the person. In a chat
eval, a DeepSeek worker redid its verification for about 5 minutes,
registered its deliverable twice, and ran out of the 15-minute attempt
clock.
> - This pull request moves that notice into the tool result that
produced the image, worded as a runner note, and removes the synthetic
user message.
> - The benefit is that a text-only agent sees a clear tool outcome and
continues its work. It does not mistake the notice for the person.

## Linked Issues or Issue Description

**What happened?**
An OpenCode agent on `openrouter/deepseek/deepseek-v4.1-flash-1017` read
a PNG screenshot that it made. The next model request contained a
user-role message with the text "Attached media from tool result: ERROR:
Cannot read image (this model does not support image input). Inform the
user." The model reasoned "The previous input from the user was ..." and
repeated finished work.

**Steps to reproduce**
1. Run OpenCode 1.18.34 with an `@ai-sdk/openai-compatible` provider and
a model that has no image input.
2. Have the model call the `read` tool on a PNG file.
3. Inspect the next chat-completions request. The last message is the
synthetic user message above.

**Expected behavior**
The model must get the notice as part of the tool result, not as a
message from the person.

**Environment**
Paperclip runner, OpenCode 1.18.34 (qualified pin), OpenRouter and
Paperclip-connection model routes.

## What Changed

- Add
`packages/paperclip-runner/src/drivers/opencode/tool-media-attribution.ts`.
It finds the OpenCode synthetic attachment message only when it directly
follows tool results. It moves each "Cannot read ..." or "empty or
corrupted" notice into the last tool result of that step as a runner
note. It keeps readable media in the synthetic message. It removes the
message when no media remains.
- The OpenCode provider proxy applies this rewrite before it forwards a
request. It re-serializes the body only when something changed.
- Messages written by a person are never changed. A person who attaches
an image to a text-only model still gets "Inform the user", which is
correct for that case.
- Add unit tests for the rewrite and an integration test through the
real provider proxy.

## Verification

- Red: the new proxy test failed before the change (`expected ... to
have a length of 4 but got 5`).
- Green: `npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts
src/drivers/opencode/tool-media-attribution.test.ts` (82 tests pass).
`tsc --noEmit -p packages/paperclip-runner` is clean.
- Real binary: I ran the real OpenCode 1.18.34 binary with a mock
OpenAI-compatible server and had the model read a PNG. The captured
request has exactly the message shape that the test fixture uses.
- `pnpm verify` returned exit 2 (test files outside the resolved
graphs). Required CI supplies broad coverage.
- A paid live run of
`chat-features.runner-opencode.local.task-coordination.delegate-with-context`
will follow in a PR comment.

## Risks

- Low risk. The rewrite matches the exact OpenCode 1.18.34 strings. If a
future OpenCode version changes them, the rewrite does nothing and the
old behavior returns. It does not fail the request.
- The rewrite is deterministic, so prompt-cache prefixes stay stable
across requests.
- No persisted state changes. OpenCode history stays as it is. Sessions
saved before this change also get the rewrite when they resume.
- Providers that do not use the runner provider proxy are not covered by
this change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:11:50 -06:00
DottaandPaperclip b2a7d322ed test: preserve runner commit stall acknowledgement invariants (#20383)
Assert that stalled event commits do not advance acknowledgement or committed state. Clarify the limits of historical failure attribution. Preserve the production implementation already merged in #20385.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-09 18:04:00 -06:00
Nicky LeachandClaude Opus 5.5 5858cef8f9 perf(runner): poll the provider right after an ACK releases it (#20374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner (`paperclip-runnerd`) sends provider events to the
controller over one authenticated socket
> - Its main loop reads one control frame, then polls the provider only
after a read finds no frame
> - After the controller ACKs the last sent event, the next read waits
the full 250 ms `RUNTIME_READ_TIMEOUT` before the Runner polls the
provider again
> - A Codex turn start needs one or two such rounds before the Runner
reads `turn/started`, so every Codex turn waits 250 ms or 500 ms for no
reason
> - This pull request reads only the frames that are already queued
after a consumed control frame, and then polls at once, when this
connection produced the unacknowledged output
> - The benefit is 250 ms less time to `turn/started` for each ACK
round, and faster event delivery after each ACK

## Linked Issues or Issue Description

This is an enhancement. The fields follow
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**
Codex turns wait for the Runner read timeout before the Runner sends
`turn/started`.

**Subsystem affected**
The Runner (`paperclip-runnerd`) durable loop in
`packages/paperclip-runner/runner/crates/runner-core/src/durable/`, for
the local Codex provider.

**Current behavior**
On a hosted test deployment (48 h, 114 Codex turns),
`runner.transport.started_event` for Codex has two modes:

| Codex turn | n | `started_event` p50 |
| --- | --- | --- |
| no provider cold start in the trace | 57 | 468 ms (all < 700 ms) |
| provider cold start in the trace | 57 | 918 ms (51 of 57 ≥ 700 ms) |

For Claude (ACPX) turns, the p50 is 90 ms. ACPX returns `turn.started`
with the command result. Codex reads `turn/started` from the provider in
a later poll.

**Proposed behavior**
The Runner polls the provider as soon as the controller ACK releases the
poll gate. The time to `turn/started` then follows the controller round
trip, not the read timeout.

**Reason and benefit**
- Real Codex (`codex app-server` 0.157, local) sends `turn/started` 2–16
ms after its `turn/start` response. The provider is not the cause.
- `runner.transport.started_event` starts when the `turn.start` command
result arrives. It ends when the controller commits `turn.started`. In
this time the Runner sends `turn.accepted`, waits for its ACK, waits 250
ms, polls, and sends `turn.started`.
- On the first turn after a provider launch, the first poll returns only
the launch diagnostic that the executor retained. A second ACK round and
a second 250 ms wait come before the poll that reads `turn/started`.
This is the difference between the two modes above (about 450 ms: 250 ms
wait and one event round).
- So 250 ms (warm turn) or 500 ms (first turn) of each Codex
`started_event` is the read timeout. The rest is one or two controller
event rounds (about 200 ms each on that deployment), less than 20 ms of
Codex time, and the 10 ms pump interval of the TypeScript transport.
- The expected benefit is 250 ms less per warm Codex turn start and 500
ms less for the first turn after a provider launch, plus faster Codex
event delivery after each ACK.

**Breaking changes**
None. No protocol, schema, or controller change.

## What Changed

- `AuthenticatedTransport::receive_queued_json` reads with a 1 ms
timeout and then restores `RUNTIME_READ_TIMEOUT`.
- New `CommandExecutor::provider_poll_is_live` hook (default `false`).
The local Codex executor returns `true` only when it owns a running
provider with no open startup attempt, so a poll cannot restore or
launch a provider. The native executor forwards it for the local Codex
facade only.
- In the steady-state loop, after a consumed control frame, the Runner
uses `receive_queued_json` when it has sent output that this connection
produced (a command or a provider poll) and the executor reports a live
provider. If no frame is queued, it polls the provider at once.
- When a provider poll returns no output, or when no frame was consumed,
the Runner uses the 250 ms read as before. A 1 ms read follows only a
consumed frame, so an idle loop does not spin.
- The replay boundary is the highest source sequence before the
welcome's pending commands run. A fresh command delivered in the welcome
counts as this connection's output. Replayed commands only resend their
stored results.
- Replayed output from an earlier connection or process, ACPX, and the
other providers keep the 250 ms read. The controller can still send its
next command before a poll restores a provider. (Without the
live-provider gate, six ACPX source-recovery tests in
`runnerd-acpx-source-reconciliation.test.ts` failed: the fast poll
restored the ACPX sidecar before the next recovery command.)
- Control frames that are already queued behind the ACK are still read
and executed before the provider poll. The ACK gate in
`poll_executor_events_after_controller_ack` is unchanged.

## Verification

- `cargo test -p paperclip-runner-core --lib durable::` (121 passed).
Five new tests in `src/durable/tests/ack_release_poll.rs`, included by
the transport test module like `hosted_retention.rs`:
-
`acknowledged_output_polls_the_provider_without_waiting_out_the_read_timeout`:
the provider poll follows the ACK read in less than
`RUNTIME_READ_TIMEOUT`. Without the change it fails (the poll waited
280–356 ms).
- `control_queued_behind_an_ack_runs_before_the_next_provider_poll`: a
command sent in the same write as the ACK executes before the provider
poll.
-
`welcome_command_output_polls_the_provider_without_waiting_out_the_read_timeout`:
the output of a fresh command in the welcome also releases the poll at
once. With the replay boundary taken after the welcome commands, it
fails.
- `replayed_output_ack_keeps_the_read_timeout_before_the_provider_poll`:
after a reconnect, a command sent 50 ms after the ACK of replayed output
executes before the provider poll. Without the replay condition it
fails.
-
`restorable_provider_ack_keeps_the_read_timeout_before_the_provider_poll`:
when the executor does not report a live provider, a command sent 50 ms
after the ACK executes before the provider poll.
- `npx vitest run src/live/runnerd-` in `packages/paperclip-runner` with
the new debug `paperclip-runnerd`: run with `--no-file-parallelism`.
Before the welcome-boundary fix: 576 passed, 3 skipped. After it: 575
passed, 3 skipped, and 1 failure once (`stops a lost runnerd's provider
and escaped tool job from its lineage journal (journal=true)`, a SIGKILL
timing test) while `cargo test` ran at the same time.
`runnerd-codex-transport.test.ts` then passed 293 of 293 three times
with this build, and three times with the base build.
- Local before and after measurement with the real Runner, the
TypeScript transport, and the fake Codex app-server (8 turns, 2 runs
each, debug build):

  | span | before p50 | after p50 |
  | --- | --- | --- |
| `runner.transport.started_event` | 377 ms (first turn 682–695 ms) |
119–122 ms (first turn 117–119 ms) |
| `runner.transport.turn_start` | 462–471 ms (first turn 775–780 ms) |
204–212 ms (first turn 247–251 ms) |

- The saving on the hosted deployment is not measured. From the analysis
above, the expected saving is about 250 ms for a warm Codex turn and
about 500 ms for the first turn after a provider launch.

## Risks

- Only local Codex gets the fast path. ACPX streaming keeps the old 250
ms wait after each ACK. A later change can opt ACPX in when its poll can
prove that it does not restore the sidecar.
- More provider polls during Codex streaming: a poll can now follow each
ACK at once instead of after 250 ms. Batches can become smaller, so the
provider state save for each batch can happen more often. Each event
already has its own durable save.
- A command that the controller sends more than 1 ms after its ACK can
now arrive after a provider poll. This was already possible after 250
ms. The change applies only when this connection produced the output, so
the provider is already active.
- If the Runner cannot restore the 250 ms read timeout, it treats the
socket as failed and reconnects. It does not continue with a 1 ms
timeout.
- No protocol, schema, or controller change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, Honeycomb queries) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 15:38:58 -08:00
DottaandPaperclip 199670ba99 fix(accounting): prevent usage receipt stalls during streamed run updates (#20385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner commits usage receipts before it acknowledges retained
events.
> - Usage receipts and streamed run logs update the same run row.
> - A repeatable-read snapshot can become stale before the receipt
writer locks that row.
> - Continuous log writes can exhaust all accounting retries and stall
the retained event.
> - This pull request uses the existing run lock with READ COMMITTED for
standalone receipt writes.
> - Other accounting operations keep their consistent snapshot. Stall
diagnostics retain safe nested error codes.

## Linked Issues or Issue Description

Related: #20235.

**What happened?**

A hosted native run stopped on a retained usage event. Its diagnostic
reported `Error at source seq N`. The saved outbox and accounting
records locate the failing delivery boundary. Traces show repeated
accounting transactions while streamed events still updated the run row.
The historical inner exception is unavailable.

A controlled PostgreSQL reproduction proves a matching defect: a run-log
append after company validation makes the receipt writer's
repeatable-read snapshot stale. Repeated appends exhaust all four
retries with SQLSTATE `40001`. This is a demonstrated failure mechanism;
attribution of the historical incident remains an inference.

**Expected behavior**

Commit the receipt under its existing run-row lock despite concurrent
run-log writes. Observe concurrent finalization and preserve
late-receipt debt. Keep duplicate receipts idempotent and reject
conflicting or cross-company receipts.

**Steps to reproduce**

Run `server/src/services/usage-receipts-read.test.ts`. The concurrency
regression uses real PostgreSQL and the production run-log append
function. It commits a log append after company validation and before
the receipt locks the run. Before the repair, the receipt rejects with a
wrapped `40001` after four attempts. After the repair, it commits once.
The finalization variant verifies that a newly settled run receives
late-receipt debt without replacing settled state.

## What Changed

- Allow aggregate-lock-protected accounting writers to select READ
COMMITTED. Keep repeatable read as the default.
- Use READ COMMITTED only for standalone usage receipt writes. Keep the
existing run lock, immutable receipt journal, replay checks and
transactional budget-work publication.
- Add PostgreSQL regressions for concurrent log appends, concurrent
finalization, duplicate delivery and the unchanged default snapshot.
- Read the outer event-commit error and at most three nested causes for
an existing safe error code. Exclude exception messages and SQL.
- Document receipt isolation and bounded stall diagnostics.

## Verification

- Red: the production receipt writer exhausted four retries with
PostgreSQL `40001` under controlled concurrent run-log appends.
- Red: wrapped database and transport stall diagnostics returned `Error`
instead of `40001` and `ECONNRESET`.
- Green: Runner control-plane tests passed all 207 cases. All 68 server
assertions passed across receipt reads, native usage delivery and cost
accounting. Cost-accounting teardown exceeded its 30-second limit twice
on this loaded host, after all 28 assertions passed. The workspace
project retained its configured 30-second hook limit despite the CLI
override. This is a local suite cleanup failure, not a passing suite;
the same revision passed the complete server suites in CI.
- Server layered TypeScript checking passed, including source and test
layers. Runner TypeScript checking passed. Server checks cover
`ccfe3bdeb1ab88537a03fd6af9044f66863939a5`; the unchanged Runner checks
cover its parent.
- Local environment: macOS arm64, Node 26.4.0, disposable embedded
PostgreSQL. Native usage-delivery tests use the real Rust receiver.
- CI at `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`:
[build](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568428),
[typecheck](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568367),
native Runner integration, browser shards and all nine serialized server
groups passed. All 12 general server shards passed in the [full
run](https://github.com/paperclipai/paperclip/actions/runs/42714760702).
Final rollup: 59 successful checks/statuses, three intentionally skipped
checks, no pending or failed checks.
- [Greptile reviewed this revision at 5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20385#issuecomment-6753163397).
- Hosted evidence came from the task, run, saved sandbox outbox,
accounting receipts and traces. The run started fresh and did not cross
a deployment. No original agent turn was replayed.
- The hosted repair has not been deployed. The original task has not
been resumed. A fixed-revision remote-provider user journey remains
unverified.

## Risks

- READ COMMITTED changes snapshot timing for standalone receipt writes.
The run-row lock protects all mutable receipt/accounting reads and
serializes competing receipts and finalization. Other accounting
operations retain repeatable read.
- Receipt identity, replay conflict checks, company scoping, budget
publication, retry bounds and acknowledgement rules remain intact.
- No persisted format, provider identity or database schema changes.
Existing saved state needs no migration.
- Diagnostics inspect at most four objects and retain only the existing
identifier format, with an 80-character code limit.
- The reproduced race matches the incident boundary and timing, but the
original inner exception was not recorded. Rollout and live acceptance
remain necessary before incident resolution.

## Model Used

- OpenAI Codex, GPT-6 family, with code and shell tools. The exact model
ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 17:35:13 -06:00
DottaandPaperclip 0e325e8d52 fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent can save a question or approval handoff before its provider
stops.
> - A missing provider terminal notification does not prove that the
saved work failed.
> - Runner already verifies provider exit, event drain, tool settlement
and suspension.
> - This change uses that proof to settle the same saved handoff.
> - Unproven shutdown holds continuation and explains that the work is
saved.
> - Recovery must never repeat the completed work or allow overlapping
execution.

## Linked Issues or Issue Description

Refs #20330. That PR extends the Codex terminal drain window. This
change handles the separate case where no terminal arrives but the owned
shutdown can prove stopped execution.

**What happened?**

A native run saved a human handoff, accepted cancellation, and later
stopped its provider. The missing terminal notification still made the
attempt fail and show “Couldn’t finish this attempt.”

**Expected behavior**

Keep the saved handoff. Accept its yielded result after either an
authoritative provider terminal or verified shutdown of the exact
execution. If shutdown remains uncertain, retain the handoff and explain
the hold without offering a duplicate Retry.

**Steps to reproduce**

Save a governed wait, acknowledge interruption, and omit the provider
terminal. The new regression uses the real Runner binary and a
controlled Codex process. The old runtime rejects the handoff instead of
asking the existing shutdown owner to settle it.

**Deployment mode**

Native Runner with Codex. Local verification used macOS arm64 and Node
26.4.0. Hosted sandbox recovery has not been qualified.

## What Changed

- Add an optional driver shutdown-settlement operation. Runner returns
proof only after its existing stop, drain, tool settlement and
suspension barrier passes.
- Bind the receipt to the run, normalized session and turn. Persist it
with the saved wait and validate it during recovery.
- Complete the same yielded handoff after verified shutdown. Retire the
stopped session from warm reuse. Do not invent a provider terminal or
complete usage.
- Retain a named shutdown-pending condition under the existing bounded
same-run recovery policy. Preserve actual provider failures and
integrity failures.
- Show saved-work shutdown guidance in task and inbox presentation. Keep
genuine approvals authoritative and disable duplicate Retry.
- Document the completion and restart contract. The checkpoint field is
additive; older checkpoints retain the terminal-or-shutdown path.

## Verification

- Red: the new runtime regression failed against the original
implementation because verified shutdown was never requested. Server
presentation and error-classification regressions also failed before the
fix.
- Red: both inbox shutdown states showed generic text before the inbox
change.
- Green: Runner runtime, shutdown and harness-backend suites pass 301
tests. Tests cover missing terminals, failed stop proof, wrong identity,
cancellation, late proof, genuine provider failure, and restart after
receipt persistence. Recovery completes once without another model turn.
- Green: the native executor and task-presentation suites pass 1,371
tests. Execution projection, inbox and blocker presentation pass 125
tests. Two test-authoring mistakes were corrected: an integrity-error
text expectation and an assertion for a nonexistent projection field.
Their initial failures remain in local logs.
- Runner TypeScript and full server layered type checks pass. Token
gates and diff whitespace checks pass.
- Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one
disposable local workspace, one tool call, and one deliberately dropped
terminal notification. The saved file, bound settlement receipt, one
accepted yielded result, one provider turn, confirmed provider exit and
suspended Runner all passed. The tool handler and persistence port were
controlled. This does not claim a full server or hosted journey.
- The full browser question-and-answer cell stopped before execution
because OPENAI_API_KEY was absent. No model work ran in that attempt.
Hosted sandbox recovery and the full browser continuation journey remain
unverified.
- `pnpm verify` returned exit 2 because the execution-spec change
requires broad CI. Required CI will supply that coverage; a full local
repository suite was not repeated.

- Review correction in `8553da244e`: a failed trailing-event write could
hide a provider failure. A new regression reproduced red, then passed
after all ordinary drain errors retained the named pending condition. No
settlement receipt or result can be accepted after that error. The live
missing-terminal check passed again on the corrected runtime.
- The real PostgreSQL control-plane regression and three existing
governed-wait controls pass. The new case persists one accepted yielded
result, a bound shutdown receipt, and the original pending human
question without a provider terminal.
- First CI revision `7c2cb9b1`: test-program typecheck found a missing
`PrpEvent` annotation in the new fixture. The annotation is fixed and
the Runner test-program typecheck passes. Runner support also timed out
in the unchanged `registered roots remain protected even when equal to
genuine public static bytes` case after 30 seconds. One unchanged local
diagnostic passed in 3.69 seconds overall. The hosted timeout cause
remains unknown; no timeout, fixture, or assertion was weakened.
[Original failed support
job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133).

- Rebased onto the merged #20330 and current recovery changes. The
combined branch passes 1,402 executor/presentation tests. Both prompt
and 18-second provider terminals still pass. A warm session with no
terminal passes after the full 30-second drain and verified
stop/suspension. Its initial overly broad assertion rejected a
nonterminal turn event; the corrected assertion rejects every terminal
type. The rebased CI gate was cancelled before any runner or step
started; its cause is unconfirmed and it supplies no validation credit.
Final revision `b49ec5a144` starts fresh CI and review.

- Final-head CI attempt 1 failed before creating jobs. GitHub reports an
internal server error (correlation
`2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty
runner-selection outputs for the aggregate jobs. One unchanged-commit
retry was requested; the initial admission failure remains recorded. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42673159868).

- **Current blocker:** final head
`b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile
review and zero unresolved threads, but required CI is not complete.
GitHub reports an active [Actions and Checks API
incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt
2 ran only part of the full selected graph. Build, canary, Rust, sandbox
providers, admitted browser shards and other completed test jobs passed.
The E2E aggregate correctly failed because `policy` and `e2e_shards` had
`abandoned` results. Server shard 5/12 received a runner shutdown signal
at 18:36:55 UTC and its test step was cancelled; its job still reports
in progress. The shutdown cause is not established. The separate review
workflow had the same pre-execution internal-error class and passed one
bounded retry. No further retry was submitted during the incident.
Resume with the missing/cancelled coverage after GitHub recovers; do not
count the absent typecheck, Runner or support jobs as passes. [Failed
coverage
gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236),
[interrupted server
shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174).

## Risks

- A missing terminal can use one additional bounded shutdown interval.
Unknown shutdown still blocks continuation.
- Provider terminal and usage accounting keep their separate meaning.
The receipt proves stopped execution and settled tool outcomes, not
model success.
- Recovery rejects a foreign receipt, a later turn, an active turn with
a settled receipt, and an actual failed provider terminal.
- There is no database migration. Older checkpoint shapes remain
readable. Drivers without this optional proof retain a named pending
state.
- This PR does not deploy the fix or repair an existing hosted task.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and local
code execution. The exact serving model ID and context window are not
exposed in this session. Live test inference used gpt-6.1-sol.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:46:16 -06:00
Nicky LeachandPaperclip 79bf733012 fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner keeps a lifetime fence until the original ACPX provider
stops.
> - A recovered Runner must prove that fence is free before it clears
saved cleanup state.
> - Suspension and other recovery paths already wait for transient
listener holders. Recovered shutdown used one immediate probe.
> - That difference made parallel Rust tests fail after they dropped
their original listeners.
> - This pull request uses the existing bounded proof for recovered
shutdown and removes the repeated test retry.
> - The benefit is one cleanup rule, with the same fail-closed proof and
no new work on normal task dispatch.

## Linked Issues or Issue Description

Fixes: #20210

Thanks to @genius-0963 for the test-only fix proposed in #20321. That
related alternative adds a test-only shutdown retry. This change applies
the existing bounded cleanup proof in the shared shutdown path and
removes the same retry pattern from native-provider integration tests.
It also tests a single shutdown call while the original quorum is still
held.

## What Changed

- Use `await_provider_lifetime_fence` for a recovered executor with no
live session and unconfirmed provider exit. Delete the unused one-shot
helper.
- Keep the acquired quorum held through the existing durable state
write. A failed proof leaves `providerExitUnconfirmed` set.
- Give the shared timeout message wording that applies to shutdown and
suspension.
- Extend the recovery test with a worker-start signal. Hold the original
listeners until the single shutdown call is waiting, then release them
and assert successful cleanup without a provider session.
- Use the same bounded proof in the retired-quorum fixture test. Remove
the duplicate retry loop from the native-provider test helper.
- Document the bounded recovery cleanup wait.

## Verification

- macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head
`368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only
targeted checks ran locally.
- `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml
--all -- --check`: passed.
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml --locked -p
paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on
the rebased head (5.21 seconds). The initial head also passed five more
parallel-test runs (45 tests per run), with no failure.
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml --locked -p
paperclip-runner-core --test native_provider_backend acpx_`: 14 passed
on the rebased head (9.43 seconds). These tests use the real Rust
executor and controlled sidecars.
- Negative control: restore only shutdown's original one-shot proof. The
new regression fails at `shutdown must wait while the original lifetime
still holds a quorum`. Restore the fix and the affected module passes.
- Independent parent verification on current commit `368fc814f7`: the
three exact recovery, held-quorum, and retired-quorum tests passed (3
tests, 5.06 seconds). Independent source review confirmed the unchanged
three-file scope and found no actionable issue.
- The active-turn recovery test still holds a live quorum through the
five-second bound. It checks that shutdown refuses cleanup, saved state
remains unconfirmed, and the provider never starts. It then releases the
quorum and confirms cleanup.
- Standard Greptile scored current commit
`368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads.
- Initial GitHub general-server shard 7 failed in unchanged
`remote-workspace-nested-verification.test.ts`, at its 15-second wait
for a child-process import checkpoint (expected one row, got zero; both
Vitest attempts). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815).
That fixture runs Node, PostgreSQL, Git, and storage migration; it does
not run the native Runner or the changed Rust cleanup. The cause of the
checkpoint timing miss remains unknown. The focused retry passed on the
unchanged source ([successful
retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)).
This proves recovery, not the cause of the original timing miss. GitHub
kept the reusable-workflow aggregate failed after that job-only retry
and rejected both a final-gate retry and a failed-job retry. A
subsequent run-level retry materialized only the gate and aggregate
jobs; its aggregate inputs reported prior jobs as `abandoned`, so it
failed without a new test failure. A new `full-ci` label event requests
a fresh workflow graph on the same commit. The reason for those
rerun-state results is not established.
- The fresh `full-ci` run
[42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208)
then lost several runners mid-job. The maintained [runner-loss recovery
42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525)
classified all four failed jobs as runner loss: typecheck, E2E shards 2
and 8, and server shard 4. The evidence was a shutdown log or
lost-communication annotation. It retried the failed jobs together;
canceled sibling jobs were also requeued. This is runner-interruption
evidence, separate from the earlier checkpoint assertion.
- The maintained recovery attempt again failed the same
partial-checkpoint assertion in unchanged
`remote-workspace-nested-verification.test.ts`, twice ([job
129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)).
Neither checkpoint-failure job logged runner shutdown or lost
communication; their resource summaries reported no significant
constraint. The child phase at the timeout remains unknown. No further
blind retry or TypeScript change is included here. Related
startup-handshake work exists in #20339; its controlled-delay evidence
establishes a deadline-boundary defect but does not establish the phase
of these original CI failures.
- Rebased onto current master after upstream #20339 merged. That
separate change fixes a demonstrated child-startup/checkpoint-deadline
boundary. It does not prove which child phase caused the earlier
zero-checkpoint assertions. This PR still changes only its original Rust
and documentation files. Fresh GitHub CI on the new base will establish
recovery, not retrospective cause.
- Full local tests and a local release build were not run. Current-head
[PR CI
42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411),
[Runner Windows
42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803),
and [Docker Runner
42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601)
all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check
list has 59 successes and 5 intentional skips. GitHub supplied broad
typecheck, build, and optimized Rust coverage. This is new-base
recovery; the cause of the original checkpoint assertion failures
remains unproved.

## Risks

- A genuinely unconfirmed original provider can now hold recovered
shutdown for up to the existing five-second proof deadline. The proof
returns immediately when two candidate ports are free.
- The changed branch requires both no in-memory provider session and
unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch
keep their existing paths. Outer controller close deadlines remain
authoritative.
- State saved by the previous release carries forward unchanged. No
schema, launch identity, profile digest, protocol, or retry-budget
change.
- No live provider session or hosted deployment was tested. The tests
cover cleanup proof, refusal, durable state, and controlled provider
execution.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository inspection,
code editing, and targeted test execution. The exact serving revision
and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 11:36:44 -08:00
DottaandClaude Opus 5.5 4a349ea447 fix(chat): tell agents that instruction saves keep no history (#20328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents can read and edit other agents' instruction files with
dedicated tools
> - #14420 changed instruction saves: Paperclip now keeps only the
current file and records no revision history
> - The agent-facing descriptions still describe a "canonical
instruction revision" history
> - In a live eval, a chat agent asked to undo an instruction change
found an empty history, read it as a test artifact, guessed the earlier
text from the current file's wording, wrote it, and reported a restore
> - This pull request tells agents the real contract in chat guidance
and in the tool and API descriptions
> - The benefit is that an undo with no saved version leaves the
instructions alone and asks the person, instead of inventing an earlier
version

## Linked Issues or Issue Description

Refs #14420
Refs #19515 (open: would list save records from the activity log. Those
records hold no content, so they still cannot restore earlier text)

**What happened**

A person asked a chat agent: "Undo Morgan's last instruction change."
Morgan's history was empty, because saves do not record revisions. The
agent's reasoning said the history "is only populated with canonical
revisions" and that there was "no revision tracking in this e2e
harness". The agent then removed the last line of AGENTS.md and replied
"Undone. Morgan's AGENTS.md is back to …". No saved source had the
earlier text, so the agent guessed it.

**Expected behavior**

When there is no saved earlier version, the agent leaves the
instructions unchanged. It says plainly that no earlier version was
saved, and it asks the person for the text they want or which part to
remove.

**Steps to reproduce**

1. Create an agent with an AGENTS.md file. Edit the file once through
`PUT /api/agents/{id}/instructions-bundle/file`.
2. `GET /api/agents/{id}/instructions-bundle/history?path=AGENTS.md`
returns `{"revisions":[]}`.
3. Ask a chat agent to undo the agent's last instruction change. The
agent rewrites the file from a guess.

## What Changed

- `AGENT_CHAT_DIRECTIVE` now says three things:
  - Saves keep only the current instruction files.
  - An empty history means no earlier text is retained.
- An undo with no listed revision must leave the file unchanged, say
that no earlier version was saved, and ask for the wanted text.
- The `get_agent_instruction_history` contract now says that it lists
only pre-upgrade revisions, and that an empty list means Paperclip
retains no earlier version.
- The `restore_agent_instructions` contract now says that only listed
pre-upgrade revisions can be restored.
- The OpenAPI summary for `GET
/api/agents/{id}/instructions-bundle/history` says the same.
- The semantic tool contracts, the protocol manifest and the seeded eval
catalog hash are regenerated.
- Regression tests check the new guidance and both live tool contracts.

## Verification

- `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts
server/src/__tests__/board-access-denial.test.ts
server/src/__tests__/codex-local-execute.test.ts
server/src/__tests__/runner-api.test.ts` passes.
- In `packages/paperclip-runner`, `pnpm exec vitest run
src/semantic-tools/paperclip-semantic-tools.test.ts src/catalog
src/evals/native-execution.test.ts` passes.
- `node scripts/generate-semantic-contracts.mjs --check`,
`generate-protocol-manifest.mjs --check` and
`generate-replay-goldens.mjs --check` pass.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- The live result is the chat eval `agents.restore-instructions`, run
with this change cherry-picked: run 42666621831 passed. The agent
replied that Morgan's history is empty and no earlier version exists. It
left AGENTS.md unchanged and asked for the exact text or the line to
remove. Run 42660236086, without this change, failed: the agent guessed
and rewrote the file.

## Risks

- Low risk. Only agent-facing text changes, and no behavior changes.
- The runner semantic catalog hash changes. This is the same kind of
change as #19983.
- If #19515 lands, history lists save records without content. The
guidance wording must then say that listed saves without content cannot
be restored. Whichever PR merges second must reconcile the two texts.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:23:12 -06:00
DottaandPaperclip 76254ead22 fix(runner): allow Codex interruption to settle governed waits (#20330)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - A native run must stop provider work before it waits for a human
response.
> - The Codex transport allows 30 seconds for a control command.
> - The governed-wait loop used its 15-second default for Codex.
> - A valid interruption terminal at 18 seconds therefore failed the
run.
> - This change gives Codex the existing 30-second terminal drain
allowance.
> - The run still fails if it cannot prove that the provider turn ended.

## Linked Issues or Issue Description

**What happened?** A Codex run can save a human wait, accept an
interruption, and fail before the provider sends its terminal. Its saved
wait remains intact, but normal finalization does not occur.

**Expected behavior:** Allow terminal delivery through the transport's
bounded command window. Keep missing-terminal cases closed.

**Steps to reproduce:** Run the new protocol regression against the old
backend. The real Runner binary uses a controlled Codex process that
delays its interruption terminal for 18 seconds. The run fails at the
15-second drain deadline. The same process passes with this change.

**Paperclip version or commit:** Reproduced against `1b7fe33432`.

**Deployment mode:** Local test of the native Runner path. The incident
was observed in a hosted sandbox.

**Installation method:** Source checkout.

**Operating system:** macOS arm64 for local verification. Hosted
incident runtime was Linux.

**Additional context:** No matching repair was found. This addresses a
reproducible deadline mismatch. The hosted log ends terminal observation
at 15 seconds, so it cannot prove whether that provider would have
settled within 30 seconds. The approved local live-provider check
passed. This PR does not claim recovery of the hosted task.

## What Changed

- Give Codex a 30-second governed-wait terminal drain through the
existing backend option.
- Keep other providers and the caller's shorter execution bound
unchanged.
- Add real Runner protocol cases for prompt, delayed, and absent
terminal delivery. Check retained wait state and one provider turn.
- Document the bounded wait and terminal-proof requirement.

## Verification

- Red: the 18-second controlled-provider case failed with
`governed_wait_provider_terminal_unproven` on the old backend. The
prompt case passed.
- Green: 282 tests passed across the native runtime, harness backend,
and new Runner protocol suite, including a repeat on the rebased PR
revision. `pnpm verify --explain` returned exit 2 because the
execution-contract document requires broad CI; the [required CI
run](https://github.com/paperclipai/paperclip/actions/runs/42668235142)
passed on the final head. All 58 applicable checks passed; the three
skipped checks were the two unrelated Storybook jobs and the
selected-test planner replaced by broad coverage.
- The missing-terminal case still fails without `completeRun` or a
successful run terminal. The saved governed wait remains available. Only
one provider turn and one tool call occur.
- Existing runtime tests cover shorter deadlines, foreign or unproven
terminals, complete usage without a terminal, and checkpoint recovery
without another model turn.
- Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real
Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64,
disposable local workspace. One `request_human_input` tool call returned
a pending interaction fixture. The real provider emitted
`turn.interrupted`; the controller accepted the yielded wait and its run
terminal. The check passed in 8.14 seconds. The tool callback and
persistence port were controlled; no hosted server permission or
Notebook publication claim is made.
- Initial local probe setup failed before model execution: first the
explicit launch lacked `app-server`, then it omitted the Runner
permission configuration. The successful check used the Runner's normal
launch arguments and isolated home with the approved existing account.
- Hosted sandbox acceptance and recovery remain unverified. The
18-second delayed shutdown is a controlled protocol regression; the live
model check proves normal shutdown, not that specific latency.

## Risks

- A Codex provider that never stops can occupy the settlement window for
up to 15 additional seconds. It still fails closed and is quarantined.
- No persisted schema, provider identity, permission, or recovery rule
changes. Previous-release checkpoints use the same format and existing
recovery checks.
- This change cannot fix a provider that never emits a terminal. It also
does not grant missing setup permissions or decide pending approvals.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed in this session. Used reasoning, tool calls, code
edits, and local execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 12:11:47 -06:00
DottaandClaude Opus 5.5 97188f276c perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.

## Linked Issues or Issue Description

Refs #20190, #20285

**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).

**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.

**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.

## What Changed

- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
  - New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
  - The other `DurableRunnerConfig` literals set `turn_scoped: false`.

## Verification

- `cargo test -p paperclip-runner-core`:
  - `--lib`: 602 passed.
  - `--test acpx_park_preparation`: 22 passed.
  - `acpx_park_readiness`: 8 passed.
  - `acpx_warm_attachment`: 4 passed.
  - `acpx_provider_suspend`: 5 passed.
  - `acpx_provider_session`: 12 passed.
  - `acpx_provider_turns`: 29 passed.
  - `native_provider_backend`: 47 passed.
  - `durable_recovery`: 3 passed.
  - `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(7fe9345030); the other arm is this branch. Host load average was about
6–9.

| | Cold first activity p50 | Cold TTFT p50 | Warm first activity p50 |
Warm TTFT p50 | Result-accepted → exit |
  |---|---|---|---|---|---|
  | Control | 5.61 s | 8.86 s | 4.95 s | 5.00 s | 1.49–1.84 s |
| This branch | 6.72 s | 9.83 s | 5.52 s | 5.57 s | 1.45–1.63 s (one
cold run 3.64 s) |

- At this load the difference is within run-to-run noise. Both arms
passed 6/6 samples.
- A process sampler saw no short-lived provider process in either arm.
Every provider process lived 11–16 s, about one run. So at load ~8 the
per-turn Runner exited before the turn-end restart ran, and this change
had nothing to remove.
- The restart did run, and fail, in the earlier A/B at load ~17–27
(#20285). This change removes that path for per-turn Runners. I did not
reproduce that load level here, so this PR has no measured speed gain to
claim.

## Risks

- **Low risk.** The change only removes work in per-turn Runners. Warm
Runners and launches without `--lifecycle-mode` behave as before.
- If a per-turn run starts a second turn, that turn restarts its
provider at its start, as before #20190. That costs the respawn on its
critical path; per-turn runs normally have one turn.
- Remote launchers that build their own `runnerd` arguments and do not
pass `--lifecycle-mode` keep today's restart. That is the safe default.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:50:25 -06:00
DottaandClaude Opus 5.5 fd2e7482c0 fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop

## Linked Issues or Issue Description

Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).

**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.

**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.

**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.

**Paperclip version or commit**
master at 4c15eea770

## What Changed

- `process_lineage.rs`: runnerd journals each supervised process's
lineage to `<runner state>/process-lineage/lineage-<runnerd
pid>-<n>.json`. The journal holds the root, every member, and every
group and session a member leads, each with its start time. The first
record is written when the provider is spawned, before any tool can run.
Later records are written by the existing 50 ms sampler only when the
record changes. A teardown that proves the lineage gone removes the
journal.
- `process_lineage/journal.rs` (new module): the on-disk format, the
write rules and the reaper `reap_journals`. The reaper refuses when
runnerd never journalled (no writer marker) or when runnerd is still
alive. It restores each journal into the same `LineageState` rules,
sends TERM and then KILL only to processes whose PID and start time
still match, and reports `verified` only when none remain. A journal
that it cannot read refuses the proof.
- Unsampled escapes: each record stores `sampled_through`, the newest
process start that its read listed. The value is refreshed at most once
a second while the membership is unchanged. The reaper refuses
(`unattributed_orphan`) when a same-user orphan (parent PID 1) that is
in no journal started after that point. Such a process can be a tool
that started after the last read, so the reaper neither signals it nor
ignores it.
- A failed journal write removes the writer marker. Removal needs no
free space. After that, no reap can verify from a record that may be
missing a member.
- `paperclip-runnerd`: durable mode enables the journal before any
provider starts. A new `reap-lineage --state-dir --runner-pid` mode
prints the report and exits 0 only when verified.
- `runnerd-codex-transport.ts`: close samples whether runnerd exited
unsuspended before close sent any signal. In that case, for a runnerd
that this transport spawned on this host, it runs the reaper from the
same executable. A verified report ends the session as a crash stop with
its durable record intact. Recovery then continues the run. Any other
result keeps the existing `NativeSessionCloseUnrecoverableError`
quarantine. Launcher-owned (remote) and adopted runners do not use this
path.
- `runnerd-codex-transport.ts`: close does not queue a durable
`runner.suspend` for a runnerd that it already knows is lost. Before,
that suspend stayed pending in the saved control-plane state. A later
runnerd on the same root then failed at startup with `terminal command
result acknowledgement changed its durable identity`. The shared
suspension barrier is unchanged for remote and adopted runners.
- `runnerd-codex-transport.ts`: a runnerd that exits during close (for
example because its provider snapshot is unreadable) is not a loss. It
keeps the quarantine.
- `runnerd-codex-transport.ts`: `#runnerHasExited` treated a
signal-killed runnerd as alive, because `exitCode` is `null` and only
`signalCode` is set. Close then spent its whole deadline trying to
suspend a dead runner. It now checks `signalCode` too.
- `fake-codex-app-server`: `--spawn-escaped-tool-job <pidfile>` starts a
tool job in its own session at turn start.
- `doc/execution-semantics.md`: describes the journal, the reaper and
the fallback.

## Verification

- Rust, `cargo test -p paperclip-runner-core --lib process_lineage` (14
pass) and `process_supervisor` (20 pass). New cases:
- `reaping_refuses_an_orphan_born_after_the_last_sample`: a tool starts
after the last read, starts a new session and exits, and leaves a
non-leader orphan job. The reap refuses and does not signal the job. It
fails without the guard.
- `a_failed_journal_write_withdraws_the_writer_marker` and
`an_unchanged_record_is_rewritten_only_to_advance_sampled_through`.
- `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`: a
real provider and a real `setsid` tool job, with a dead writer. The reap
stops both, proves them gone and retires the journal and marker.
- `reaping_refuses_without_a_marker_or_while_the_writer_lives` and
`reaping_refuses_an_unreadable_journal`.
- `a_journalled_record_restores_the_same_lineage`: after a round trip,
the restored record still finds a re-parented job and its later child,
and ignores an unrelated process.
- TypeScript, live with the real debug runnerd: `stops a lost runnerd's
provider and escaped tool job from its lineage journal`.
- `journal=true`: SIGKILL runnerd mid-turn. The escaped tool job
outlives runnerd. Close resolves, the job is gone, and the report is
`verified`. Before the fix, close rejected with `runner did not durably
suspend before checkpoint` and the job stayed alive.
- `journal=false` (an older runnerd with no journal): close still
rejects, and the job is not signalled.
- `runner-lineage-reap.test.ts`: a report from an older runnerd, a
malformed report, or an inconsistent `verified` report never counts as
proof.
- Turn-start cost: the journal adds one single-process read and one
small file write when runnerd spawns its provider (about 1.5 ms in a
macOS debug build), once for each runnerd. Nothing is added for each
turn.
- Paid runner E2E,
`chat-features.runner-opencode.local.recovery.retry-exhausted`, on the
eval branch with this fix (runs 42637596571 and 42644400434). Before the
fix (run 42615758956), attempt 2 stopped with
`native_session_cleanup_quarantined`. Now each of the three runnerd
kills records `native_runner_process_exited`. Attempts 1 and 2 are
`retryable_failure`, and attempt 3 is `terminal_failure` with cause
`native_session_retry_exhausted` (attempt 3 of 3). The reaper ran after
the first kill and verified. The cell still fails one eval setup check.
The check waits for an *active* exhausted recovery action, but
`execution-recovery-resolution.ts` settles that action as resolved and
blocks the task (`preserve_without_replay_v1`). That is a separate
grader decision, outside this PR.

## Risks

- Medium. The change only affects a close where runnerd already exited
without suspending and without a signal from close. Every other close is
unchanged.
- The reaper signals processes by PID. Each signal re-checks the PID's
start time first, and a group or session is used only while its leader
PID still names the recorded process (the same rules as the existing
in-process teardown).
- A crash stop leaves the durable root unsuspended. The coordinator
already scheduled a retry for `native_runner_process_exited`. That retry
now runs instead of failing on the quarantine. The paid E2E shows that
attempts 2 and 3 start on the same run.
- The escape guard can refuse when an unrelated same-user process
becomes an orphan in the window of about one second after the last read.
The fallback is today's quarantine. An orphan of a subreaper other than
init is not seen by the guard.
- The journal adds at most one small write each second for each runnerd
while processes start on the host. This happens on the background
sampler, not before turn start.
- Previous-release story: no database or saved-session schema change.
The journal is a new file in the runner state directory. A runnerd from
an older release writes no journal and has no `reap-lineage` mode. Its
loss keeps today's quarantine, and the existing board Retry and
stopped-session proofs still apply.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:19:41 -06:00
DottaandClaude Opus 5.5 e015c2ef21 fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path

- #20145 prepares the next Claude provider generation at park, and
#20190 restarts the ACPX provider when a turn settles. They move the
sidecar respawn off the follow-up's critical path.
- When that turn-end restart fails, the runner "fails closed":
`restart_session_after_turn` saves the session `closed` and drops it.
Within the run, the next turn fails exactly like a failed restart at
turn start. That's intended.
- Across runs it breaks the conversation:
- the next run's `run.attach` refuses a `closed` session ("run.attach
requires the same settled ACPX provider profile and session");
  - it reconnects for about 10 s;
- the controller then records a continuity break and starts a fresh
Claude conversation (`run.startup.reuse provider_session:
fresh/created`).
- A local A/B (`ttft-baseline --adapters claude --samples 3 --phases
coldwarm`, on a host with load average about 17–27) confirmed it:

  | | d40a7494 (before #20145/#20190) | a3a25ce6 (after) |
  |---|---|---|
| Follow-ups that resumed the conversation | 3 of 3 (`verified_reuse`,
resume 4.4–8.2 s) | 1 of 3 |
| Follow-ups that started a fresh conversation | 0 of 3 | 2 of 3
(bootstrap 16–17 s) |

The kept instance shows each lost follow-up's runner received provider
state `lifecycle: "closed"`, with `run.attach` refused.
- A failed park preparation already recovers safely: it waits for the
reaped provider's lifetime fence, then saves `suspended`. The turn-end
restart should do the same.

## Linked Issues or Issue Description

Refs #20190, #20145

**What happened?**
When the post-turn ACPX provider restart failed, the next Claude run
could not attach. It spent about 10 s and then started a new
conversation, losing the task's conversation context.

**Steps to reproduce**
1. Run a Claude task on a busy host so the turn-end provider restart
fails (the fixture `reject-open` fault reproduces it).
2. Post a follow-up comment (a new run).
3. The follow-up's `run.attach` is refused, then `run.startup.reuse`
reports `provider_session: fresh/created`.

**Expected behavior**
The follow-up resumes the same Claude conversation. Within the failing
run, later turns still fail closed.

## What Changed

- `acpx_park_preparation.rs`:
- A failed `restart_session_after_turn` now calls the shared
`recover_failed_generation_restart`, renamed from
`recover_failed_park_preparation`.
- That recovery drops the session, marks it `prepared` with the provider
exit unconfirmed, waits for the provider-lifetime fence, then saves
`suspended`.
- If the fence or the save fails, the state stays exit-unconfirmed,
which is the safe direction.
- `acpx_provider_backend.rs`:
- Later turns of the failing run still fail closed: the first with "ACPX
turn start failed closed: <cause>", then "ACPX session is not open".
That's the same as a failed restart at turn start.
- The restriction is saved in the ACPX state as
`turnEndRestartFailedRun` (serialized only while set), so it outlasts a
Runner process restart inside the same run. Same-run restore doesn't
reopen the session while it's set.
- Only a `run.attach` from a different run clears it; a same-run attach
keeps it. State validation requires it to match the descriptor's run.
- The cause is handed to the durable Runner once
(`take_turn_end_restart_failure`), which records it as a redacted
`turn-end provider restart failed: <cause>` diagnostic in
`runner-state.json`. That gives the under-load cause a first recorded
sample.
- Tests (`tests/support/acpx_turn_end_restart.rs`):
- New:
`the_next_run_resumes_the_conversation_after_a_failed_turn_end_restart`,
red on master (the state is `closed`). It covers the same Runner process
with a warm attach, and a new Runner process with a cold attach. The
conversation id is preserved, and the next run completes a turn.
- New:
`a_runner_restart_in_the_same_run_keeps_the_failed_turn_end_restart_closed`,
red without the saved field. A new Runner process with the same run id
fails `turn.start` without spawning a provider; then a new run
cold-attaches and resumes the same conversation.
- The existing
`a_failed_turn_end_restart_fails_the_next_turn_like_a_failed_turn_start_restart`
now also asserts that the turn-end failure saves `suspended` (the
turn-start failure stays `closed`). Everything else about the run's
failing turns is unchanged.

## Verification

- `cargo test -p paperclip-runner-core`:
  - `--test acpx_park_preparation`: 21 passed.
  - `--lib acpx`: 93 passed.
  - `acpx_park_readiness`: 8 passed.
  - `acpx_warm_attachment`: 4 passed.
  - `acpx_provider_suspend`: 5 passed.
  - `acpx_provider_session`: 12 passed.
  - `acpx_provider_turns`: 29 passed.
  - `native_provider_backend`: 47 passed.
  - `durable_recovery`: 3 passed.
- Live A/B on this branch (`ttft-baseline --adapters claude --samples 3
--phases coldwarm`): follow-ups resumed 3/3 (1/3 before), follow-up
first activity p50 6.2 s (about 20–23 s before).
- `cargo clippy --all-targets`: no findings on changed lines.

## Risks

- **Late-callback guarantee:** unchanged. The failed restart already
reaped the old generation, and the next generation starts only via
`run.attach` or a later reopen. That's the same resume path a failed
park preparation uses.
- **Fence wait:** the failure path now waits for the reaped provider's
lifetime fence before saving, matching park recovery. If the fence never
clears, the state stays exit-unconfirmed and fails closed.
- **Rollback:** `AcpxDurableState` denies unknown fields. The new field
is written only between a failed turn-end restart and the next run's
attach. A rollback inside that window makes the older Runner refuse the
saved state, which still fails closed, as master's `closed` state does
today.
- **Scope:** the cause of restart failures under load isn't changed
here. This makes their failure mode safe.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:15:25 -06:00
Nicky LeachandPaperclip 09412e652a fix(runner): allow warm release to drain after provider restart (#20290)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Warm Runner sessions park after a turn so follow-up tasks can claim
the same process.
> - Claude restarts its ACPX provider after the controller receives the
terminal result.
> - That restart can delay the drain receipt beyond the release path's
one-second limit.
> - A failed drain prevents parking, so the idle timer later closes the
retained Runner.
> - This pull request uses the existing warm admission budget for
release drain.
> - The benefit is retained Runner reuse without moving provider restart
onto task dispatch.

## Linked Issues or Issue Description

Fixes: #20225. Refs #20190 and #20145. Related performance follow-up:
#20224.

## What Changed

- Align the release drain timeout with the existing 30-second warm
admission budget for every provider. Close paths keep their explicit
remaining deadlines.
- Keep the existing immediate completion when the drain receipt proves
settlement. Add no sleep, retry, provider branch or dispatch work.
- Add a real Runner regression with a fake ACPX sidecar whose
replacement start takes 1.5 seconds. One release call must park the
Runner, then a follow-up must claim the same Runner process.
- Document the release drain budget.

## Verification

- Targeted command in `packages/paperclip-runner`: `vitest run
src/live/runnerd-codex-transport-release-resume.test.ts
src/live/runnerd-codex-transport-settlement.test.ts
src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed**
across 3 files, 157.78 seconds.
- Independent parent verification reran the slow-restart and ordinary
cached cases: **2 passed**. Source review confirmed that release is the
only caller using the changed default; close callers supply explicit
deadlines.
- Negative control: with the original one-second timeout, the new test
fails because the single release call returns `false`. The fixed code
passes the new case.
- Built only the current-source Runner and two fixture binaries required
by these tests. The first sandboxed invocation could not bind its
loopback listener (`EPERM`); the test ran with loopback access
afterward.
- Full GitHub PR workflow
[42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993)
passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard
Greptile scored this same head 5/5 with no review threads. Full
repository checks ran in GitHub CI, per the requested local test scope.
No live Claude session or hosted deployment was tested.

## Risks

- An unresponsive Runner can now hold release drain for up to the
existing 30-second warm admission limit. A healthy Runner returns as
soon as its receipt arrives. The provider restart stays at turn end;
task dispatch gets no fixed delay.
- Saved state, protocol and provider identity formats do not change.
Previous-release state carries forward under the existing validation.
- The controlled fake-sidecar test proves the restart timing race and
process reuse. It does not qualify live provider latency or a hosted
rollout.

## Model Used

- OpenAI GPT-6 through Codex, with repository tools, code editing and
targeted test execution. The exact serving revision and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 07:43:27 -08:00
DottaandClaude Opus 5.5 3374ca54f5 fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (`runnerd`) runs each provider turn, and the
server records why a run stopped and what the person can do next
> - When `runnerd` exits during a provider turn, the server saved the
run as `native_provider_terminal_failed`: "The provider session is
permanently unusable", with automatic retries stopped
> - The real cause was the runner process, and the transport already
names it (`native_runner_process_exited`) at the start of the
closed-stream message
> - Separately, the process-lineage sampler added in #19727 panicked on
Linux debug builds when a provider tool started its own session, and
that panic text became the run's visible failure
> - This pull request records a runner exit mid-turn as runner loss, and
reads session IDs without the assertion that panicked
> - The benefit is that a person sees the true cause of a stopped run,
and recovery treats a lost runner like every other runner exit

## Linked Issues or Issue Description

Refs #19727

**What happened**

A chat eval killed only the owned `runnerd` process (SIGKILL) while a
provider tool was running. The run failed with `errorCode:
native_provider_terminal_failed` and the next action "The provider
session is permanently unusable". The error text was `Notification
transport failed closed: native_runner_process_exited: runnerd exited
unexpectedly: paperclip-runnerd panic: panicked at
rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came
from the lineage sampler thread before the kill. It called
`rustix::process::getsid`, which asserts a nonzero session. A kernel
thread, or a process whose session leader is outside the PID namespace,
reports session 0.

**Expected behavior**

A runner process exit is recorded as `native_runner_process_exited`, the
same cause the server already uses for a runner exit outside a turn.
Recovery then applies its normal runner-loss rules (checkpoint evidence,
bounded attempts, cleanup quarantine). The lineage sampler does not
panic, and its diagnostics do not replace the real failure text.

**Steps to reproduce**

1. Start a native OpenCode run on Linux with a debug `runnerd`.
2. Let the agent run a shell tool that starts its own session (for
example a long `node -e` command).
3. Send SIGKILL to the `runnerd` PID only.
4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is
`native_provider_terminal_failed` and the error names the rustix panic.

## What Changed

- `nativeSessionFailureSourceCode` returns
`native_runner_process_exited` for a `NativeProviderTerminalFailure`
with code `notification_transport_failed` when its message starts with
`Notification transport failed closed: native_runner_process_exited: `.
Only the runner transport writes that prefix. Other closed-stream causes
stay `native_provider_terminal_failed`.
- `paperclip-process-table` has a new `session(pid)` function. Linux
reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A
session of 0 or an error gives `None`.
- `process_lineage` reads sessions through
`paperclip_process_table::session` instead of rustix `getsid`.
- Regression tests: the runner-loss classification (attempts 1 and 3,
and three negative shapes), the stat session parser with a kernel-thread
line, and a live read of this process's session.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "runner
process exit that closed|preserves stable provider and runner|model
rejection from a provider terminal|operator-owned through terminal"` (8
passed)
- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests
run in CI)
- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --lib process_lineage` (6 passed)
- End to end: the `recovery.*` chat eval cells that kill `runnerd` are
run against this change on a separate eval branch.

## Risks

- Behavior shift: a runner exit during a turn now follows runner-loss
recovery instead of stopping as a permanent provider failure. Recovery
still decides from durable evidence. Ambiguous state stays terminal, and
an unverified cleanup stays quarantined.
- Previous-release story: no persisted shape changes. Runs saved earlier
with `native_provider_terminal_failed` keep that code. Only new failures
are classified with the new rule.
- The session read on Linux now uses `/proc` instead of a syscall. It
gives the same value, and it reports 0 (now `None`) in the same cases.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 06:38:24 -06:00
DottaandClaude Opus 5.5 8eb11fa8aa fix(runner): fail closed with a recorded cause when runner event commits stall (#20235)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Runner streams numbered events to the control plane, which commits
each one and acknowledges the committed prefix
> - In restart chaos wave `w-20261109T093417-s9807`, t01 (claude R1,
local), the control plane never committed Runner seq 34. Every failure
path in `#event` closed the connection without recording why. The Runner
reconnected 2,848 times over 14 minutes and nothing detected it
> - This pull request makes that loop a detected, named failure that
carries its cause
> - The benefit is that a wedged event stream becomes a visible run
failure with a reason, instead of a silent "running" task

## Linked Issues or Issue Description

Refs #19831. Restart chaos wave `w-20261109T093417-s9807` (master
`df4562ab38`), task t01, run `d08dc899`.

**What happened** (Phase 2's re-check of the kept instance):
- The provider kept working. Claude's transcript runs to 09:47:03 and
ends in `paperclip_finish`.
- The Runner, its ACPX provider state, and the controller journal agree
on acked seq 33. The controller never commits Runner seq 34 (Write
progressed, 09:36:25.892).
- Phase 2 replayed the kept outbox offline. Runner seq 34 fails
`validatePrpEvent`: its `/payload/target` contains a newline from a
malformed Claude Write call. `#event` closes without a reason, and the
Runner redelivers the same bytes forever. (Phase 2's separate PR stops
the producers from emitting such paths.)
- Every `#event` failure path closed the connection without recording a
reason, so the cause was invisible. An outbox that already holds such an
event can only be cleared by failing closed.
- The `<instance>:<run>:N` ids in `heartbeat_run_events` come from the
TS driver's own projection numbering, so there was no Runner sequence
rewind.

## What Changed

-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`:
- `#eventCommitFailed` replaces the bare `connection.close()` on each
`#event` commit-failure path. It records a bounded cause:
`event_sequence_mismatch` for an out-of-order event,
`event_schema_invalid:<issue paths>` for an event that fails
`validatePrpEvent`, the error's code or name (never its message) for a
commit callback error, `journal_sequence_not_contiguous`,
`journal_commit_refused`, or `checkpoint_flush_failed:<cause>`.
- If failures continue for the same run at the same acknowledged cursor
for at least 20 attempts and 45 s, it fails closed with
`NativeSessionProtocolIntegrityError("runner_event_commit_stalled",
"<cause> at source seq N")`, where N is the failing event's own
sequence. Only a moved acknowledged cursor counts as progress (a
replayed, already acknowledged duplicate does not), and a different run
starts a fresh counter. A single transient failure still just
reconnects, as before.
- A checkpoint flush that fails before any outbound frame also counts,
as `checkpoint_flush_failed:<cause>`. That includes the authentication
challenge and the welcome, so a reconnect that cannot even authenticate
still reaches the bound.
- A journal write that throws (`#store.commit`, `#store.commitEvent`, or
the welcome save) leaves the store unwritable until reload, so it fails
closed at once with `journal_write_failed:<code>`.
- `packages/paperclip-runner/src/contracts/native-session-backend.ts`:
new reason `runner_event_commit_stalled`, with an optional `detail` in
the message.

## Verification

Red/green:
- "bounds commit-stall reconnects for an out-of-order event": a
schema-valid event with the wrong `sourceSeq` fails closed with
`event_sequence_mismatch at source seq 5`. Before this it closed
silently forever. "… for 20 failures inside 45 s" pins the time bound:
20 failures over 19 s do not fail closed.
- "fails closed when checkpoint storage keeps failing before a reconnect
authenticates": with checkpoint persistence refusing (`ECONNREFUSED`),
20 reconnect attempts fail before authentication, then the control plane
fails closed with `checkpoint_flush_failed:ECONNREFUSED at source seq
1`. Without this, those attempts close silently.
- "fails closed at once when the journal cannot record a runner event":
a journal write throwing `ENOSPC` fails closed with
`journal_write_failed:ENOSPC at source seq 1`, and the private error
text is not carried. On master the connection only closes.
- `durable-prp-control-plane.test.ts` "fails closed with the schema
issue when the runner keeps redelivering an invalid event (restart chaos
t01)": an event that fails `validatePrpEvent` is redelivered 20 times
over 57 s of mocked clock. On master the control plane only closes; with
this PR it fails closed with `runner_event_commit_stalled` and detail
`event_schema_invalid:/… at source seq 1`.
- `durable-prp-control-plane.test.ts` "fails closed when runner events
keep failing to commit without progress": 20 reconnects over 57 s of
mocked clock, each commit failing with `ECONNRESET`. On master nothing
fails closed. With this PR the 20th failure reports
`runner_event_commit_stalled` with detail `ECONNRESET at source seq 1`,
and the private error message is not carried. File: 201 passed,
including the existing "keeps a non-typed commit failure retryable"
cases.

Static checks:
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.
The server is unchanged.

## Risks

- Storage that keeps failing event commits for at least 45 s and 20
attempts now fails the run (operator recovery) instead of looping
silently. Shorter outages are unchanged.
- The recorded cause is a code or error name only, never error text, so
no tool content or credentials reach the run error.
- No schema change and no change to persisted state formats.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 05:57:46 -06:00
DottaandClaude Opus 5.5 3f2a940d5b fix(runner): answer held Paperclip tool calls after a local restart before the next turn (#20227)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart interrupts a turn, the Runner keeps each Paperclip
tool call the model made as "held" until Paperclip answers it, and it
refuses to start the next turn while any call is held
> - Paperclip answers held calls after a reattach through its tool
recovery path. That path was wired only for remote runners
> - In restart chaos wave `w-20261109T084023-s9403`, t101 (local codex,
K2 then K1) held a `read_document` call that nothing answered, so the
continuation's `turn.start` failed and the task needed a Retry
> - This pull request wires held-call recovery for local runners too,
and makes `turn.start` answer or wait for held calls first
> - The benefit is that a turn lost to a restart no longer strands the
run behind one unanswered call

## Linked Issues or Issue Description

Refs #19831. Restart chaos wave `w-20261109T084023-s9403` (master
`fc5c79a327`), task t101 (codex C1, K2, local).

**What happened:**
- Run `4b5810d5` called `read_document` (`exec-ac8fa9b4`) around the K2
drain at 09:08:18. A K1 then killed the next server start.
- At 09:08:36 the third controller reattached. The Runner restored the
conversation and reported the old turn lost
(`provider_turn_lost_on_restore`, recoverable). The held call stayed
pending, by design: a held call accepts a late result.
- The controller had the call: the quarantine detail lists
`exec-ac8fa9b4` under `semanticTools.pending`. But
`recoverPendingSemanticInputs()` returned at once: for a local run
`controllerPersistence` is unset, so `recoverSemanticToolInput` was
`undefined`.
- At 09:08:46 the continuation's `turn.start` failed with "cannot
prepare the next provider turn while tool calls are active (pending: 1
[read_document:exec-ac8fa9b4…])". The run ended
`native_session_interrupted`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: the
runnerd transport now always gets `recoverSemanticToolInput`, not only
when `controllerPersistence` (remote) is set. It uses the same epoch
check and `SessionAuthorityEpoch.recover`. Recovery re-authorizes and
re-runs reads, and replays receipts for mutations; it never repeats an
uncertain mutation.
- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`
(`#startTurnWork`): when the control plane has unsettled semantic
inputs, it runs `recoverPendingSemanticInputs()` before queueing
`turn.start`, and waits for that pass, bounded by the turn-start
deadline. Recovered results are queued as commands ahead of
`turn.start`, and the Runner applies commands in order. If recovery has
not finished by the deadline, the turn start fails with
`native_turn_start_held_tool_calls_unresolved` instead of sending
`turn.start` behind it. When everything is settled there is no wait, so
a normal turn start is unchanged. Live calls of a turn being replaced
are not waited on; the replaced start still rejects them.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`:
`recoverPendingSemanticInputs()` passes are serialized. A call made
while the startup pass is still answering a held call waits for that
pass, then runs its own, so the turn-start gate cannot finish early by
skipping a call that is already being recovered.

## Verification

Red/green:
- `native-session-executor.test.ts` "answers a local run's held
Paperclip tool calls after a restart (restart chaos t101)". On master
`recoverSemanticToolInput` is `undefined` for a local run; with this PR
it is wired.
- `runnerd-codex-transport.test.ts` "answers a held call before starting
the next turn (restart chaos t101)", against real runnerd and fake
Codex, with a slow startup recovery pass already in flight when
`turn/start` arrives. On master `turn/start` never ran recovery, and
without the serialization `turn.start` is queued before the held call's
result. With this PR the held call is recovered once and its result is
queued before `turn.start`.

Regression and static checks:
- `runnerd-codex-transport.test.ts`,
`durable-prp-control-plane.test.ts`,
`runnerd-codex-transport-adoption.test.ts`: 510 passed.
- `native-session-executor.test.ts` and `native-session-resume.test.ts`:
1368 passed, 1 failed. The failing case, "archives a damaged prior epoch
… with real runnerd", fails identically on master on this Mac (`ENOENT …
authority-epochs`), so CI is its reference.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- Local runs now recover held calls after a reattach, as remote runs
already did. Reads re-run under current authority. A mutation without a
saved receipt still returns no result and stays held, as before.
- `turn.start` may now wait up to its existing deadline while a recovery
pass runs, then fails clearly if it has not finished. It never waits
when nothing is unsettled, so normal turns get no added latency before
`turn.start`.
- No schema change and no change to persisted run state formats.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 05:26:12 -06:00
DottaandClaude Opus 5.5 465da4227a fix(runner): drop a tool target path that breaks a line so the controller can acknowledge its event (#20238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner turns provider tool calls into durable PRP
events. The controller acknowledges each event only after it validates
and commits it.
> - The controller's event schema forbids line terminators in a safe
path such as `tool.execution` `target`.
> - The ACPX sidecar and the Rust runner accepted a location path that
contained a newline. Claude sometimes emits a malformed Write call that
puts the file content into the path.
> - The controller refused that event without an acknowledgement. The
runner redelivered the same bytes, and the run stopped making progress
for 14 minutes (2,848 reconnects).
> - This pull request makes both producers drop such a path, so the
event carries a null target and the controller accepts it.
> - The benefit is that a malformed provider path can no longer wedge a
run's event delivery.

## Linked Issues or Issue Description

**What happened?**
In restart-chaos wave seed 9807 (claude, R1, local), the run stopped
receiving events after runner source seq 33. The runner reconnected
2,848 times in 14 minutes. Offline replay of the saved runner outbox
through the controller's `validatePrpEvent` showed that seq 34 fails
`/payload/target` (the safePath pattern), because the target is
`packages/shared/src/chaos/slug-2c5c3c8f.ts\n<parameter
name="content">export function chaosSlug…`. Seq 35, 39 and 40 (the same
tool call) fail the same way.

**Expected behavior**
The runner never emits an event that the controller's schema refuses. A
malformed tool path becomes a null target, and the tool event still
arrives.

**Steps to reproduce**
1. Have an ACPX provider send a tool call whose location path contains a
newline (for example, a Write call with the content in the path).
2. See that the runner emits `tool.execution.progressed` with that
target.
3. See that the controller closes the connection without an
acknowledgement on every redelivery.

**Paperclip version or commit**
master at 3963d38560

## What Changed

- `runner-core/src/tool_file_changes.rs`: new `is_single_line_path`,
which refuses control characters and U+2028/U+2029 (the characters
ECMAScript `.` does not match). `normalize_relative` uses it, so
file-change paths are covered too.
- `runner-core/src/provider_events.rs`: `safe_acpx_location` uses the
same check (it replaces the NUL-only check).
- `src/drivers/acpx/safe-locations.ts`: the sidecar drops a location
whose path contains a control character or U+2028/U+2029.
- Tests: a Rust test covers the observed path, `\r`, U+2028, U+2029 and
tab, plus a full ACPX tool-call normalization with a broken location and
file change. A TS sidecar test covers the same paths.

The controller-side detection and reason capture for a refused event is
in paperclipai/paperclip#20235 (Phase 5). This PR does not touch the
controller.

## Verification

- Red before the fix: the Rust test
`drops_a_location_whose_path_breaks_a_line` and the TS test `drops a
path that breaks a line…` both failed. Both pass after it.
- `npx vitest run src/cli/acpx-sidecar-locations.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx` in
`packages/paperclip-runner`: 1,818 passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in
`packages/paperclip-runner`: passes.
- `cargo test --lib` in `runner-core`: 598 passed. Four
`task_memory::tests::monitor_*` tests fail on this macOS host in
isolation too. That file is unchanged here (last changed in #18801), and
the CI Rust job is the authority.

## Risks

- Low. Only a path that the controller already refuses is dropped. A
valid POSIX filename with a tab is now dropped as a display target too,
which is consistent with the existing ACPX rich-event check
(`char::is_control`).
- Previous-release story: a runner outbox already saved with such an
event (from an older runner) still holds it. That case needs the
controller detector in paperclipai/paperclip#20235, which fails the run
closed instead of looping. No saved state, schema or pack contract
changes here.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 05:06:43 -06:00
DottaandClaude Opus 5.5 d7d0f96f3f perf(runner): save provider state once per poll instead of once per frame (#20217)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode, Codex and Cursor Cloud chat turns run through
runnerd. runnerd reads provider frames in order and keeps the run's
provider state in a durable file
> - For every frame that it reads, runnerd rewrote the whole provider
state file and called `fsync` on the file and on its directory. The
state holds every tool result kept in the turn, so each save got more
expensive as the turn went on
> - A tool call request is just the next frame. It waited behind one
full save for every frame before it. #20209 measured tool-call
turnaround that grew from about 1.4 s to about 15 s in one turn
> - #20209 stopped the OpenCode proxy from sending frames that runnerd
discards. Every frame that runnerd still reads still paid a full save,
and other proxies (for example Cursor Cloud) still send such frames
> - This pull request makes runnerd save the frames of one poll once,
before anything derived from them leaves runnerd. A save that would
write the bytes already on disk is skipped
> - The benefit is that the number of saves before a tool call no longer
grows with the number of streamed frames. In a local measurement, a tool
call after 400 frames with 830 KB of kept state took 0.57 s instead of
more than 30 s

## Linked Issues or Issue Description

Refs #20209
Related: #17514 (the same per-event full-state rewrite, on the server
control-plane side)

**What happened**

During a native OpenCode chat turn, tool-call turnaround grew from about
1.4 s to about 15 s as the turn went on. runnerd saved its complete
durable provider state, with `fsync`, after every provider frame. That
state includes the tool results kept so far in the turn. A tool call
request waited behind one full save for each earlier frame.

**Expected behavior**

The time to deliver a tool call must not grow with the number of
assistant-text, reasoning or item frames that came before it. A crash or
restart must still never lose state that a resumed run needs.

**Steps to reproduce**

1. Start a native OpenCode (or Codex app-server or Cursor Cloud) turn
that keeps a few tool results and then streams many text or reasoning
frames before its next tool call.
2. Measure the time from the tool call frame to the
`semantic_tool.input` event.
3. See that the time grows with the number of earlier frames and with
the size of the kept state. Locally, `cargo test -p
paperclip-runner-core --lib measure_streamed_frame -- --ignored
--nocapture` on `master` reports 102 saves and 9.3 s for 100 frames with
830 KB of kept state, and more than 30 s for 400 frames.

## What Changed

-
`packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs`
and `codex_provider.rs`:
- The provider frame loop moves into `poll_provider_frames`. An ordinary
notification frame and a descendant notification frame no longer save
the state. They mark it as unsaved.
- After the loop, `poll_current_provider_with_control` saves the unsaved
state once. It also saves when a frame fails, before it returns the
error.
- Frames that already saved at once still do so: turn terminals, Cursor
Cloud terminal deliveries, `reconciliation_required`, protocol failures,
MCP permission and runtime requests, process exit, and new tool calls.
- A tool call that runnerd answers at once (an exact completed replay or
the receipt-limit stop) saves unsaved state before it writes to the
provider. A rejected tool call already saved first.
- `retained_events`, `execute` and `host_park_ready` save unsaved state
first, so no event, command result or park decision can expose state
that is not on disk.
- Courtesy rejections that `CodexProvider::poll` writes by itself
(unsupported requests, descendant requests, delayed requests after a
terminal, full runtime-request capacity) are held while runnerd reads a
batch (`hold_courtesy_responses`). runnerd sends them only after it
saves the state that covers every earlier frame
(`save_before_provider_writes`). A terminal frame also saves and
releases them before runnerd asks the provider for goal state or drains
the accounting tail. Released courtesy replies are best-effort, as
courtesy replies were before: a provider that already closed stdin
cannot turn a received terminal into a polling failure. The
receipt-limit stop sends held cancellations before its `turn/interrupt`
request, so their order to the provider does not change.
- The trace records for runnerd's normalization
(`rust_durable_normalization`), which name emitted event ids, are kept
until the state that holds those ids is saved. The trace therefore only
names saved events. Saved mappings are written before an exited or
failed provider is dropped, so they are not lost. Raw frame records
still go to the trace when a frame is read; they name no event ids.
- `save_unsaved_frame_state` does not write when the serialized state is
byte-for-byte the same as the saved file. All other saves write as
before.
- Serialization moves to `serialize_provider_state`, and the file write
moves to `write_state_bytes`. The write path (private temporary file,
`fsync`, rename, directory `fsync`, `0600`) does not change.
- New tests in
`packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs`:
- `streamed_frames_do_not_multiply_durable_saves_before_a_tool_call`: 8
and 48 streamed frame pairs before a tool call need the same number of
saves (1). On `master` they need 18 and 98.
-
`batched_frame_state_is_saved_before_events_leave_and_survives_a_restart`:
at every poll, the saved file is equal to the in-memory state and holds
exactly the delivered events. A new executor loads the file after the
old one stops and gets the same state, the same unacknowledged events
and the pending tool call.
- `batched_frame_state_is_saved_when_a_later_frame_fails`: a frame that
fails after 20 batched frames still leaves those 20 frames on disk.
- `previous_release_state_is_rewritten_by_the_first_batched_save`: a
state file in an older shape (compact JSON without newer default fields)
loads, and the first batched save rewrites it. Unchanged state is not
rewritten. Changed state is.
-
`courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved`:
when the courtesy rejection reaches the provider, the saved file (what a
restarted runnerd would load) already holds the 20 earlier frames and
equals the in-memory state. It fails if the hold is removed.
- `trace_interpretations_name_only_saved_events`: every event id that a
trace record names is in saved state at the moment the record is
written. It fails if records are written before the save.
-
`a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`:
after 122 frames are read and not yet saved, the saved file is
byte-for-byte unchanged and runnerd has written nothing to the provider.
A reload gets the pre-batch event sequence and no pending events.
-
`a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`:
the provider closes stdin and then completes the turn with a tool call
still pending. The held cancellation cannot be written, and the turn
still completes and is saved.
- `trace_interpretations_survive_a_provider_exit_in_the_same_poll`: the
trace keeps the mappings for frames that arrived in the same poll as the
provider exit.
- `measure_streamed_frame_cost_before_a_tool_call` (ignored): prints
saves and tool-call latency for different frame counts and kept-state
sizes.

## Verification

- `cargo test --locked -p paperclip-runner-core` (debug, macOS arm64):
library 595 passed, 1 failed; all 35 integration test binaries passed
(`--no-fail-fast`). The one library failure is
`task_memory::tests::monitor_bounds_notices`. It also fails when run
alone, takes about 9 minutes, and waits on file-watch timing in
`task_memory.rs`, which this PR does not touch. In one earlier run, two
ACPX park tests in `acpx_park_preparation` failed. The same binary also
failed on unchanged `master` (with a different test), and it passed in
the final run.
- `cargo test -p paperclip-runner-core --lib measure_streamed_frame --
--ignored --nocapture`, debug build, macOS APFS (`sync_all` is
`F_FULLFSYNC`). Each row is one turn: frames streamed, then one tool
call.

| Kept state | Frames | Saves (master) | Tool-call latency (master) |
Saves (this PR) | Tool-call latency (this PR) |
| --- | --- | --- | --- | --- | --- |
| 4 KB | 100 | 102 | 946 ms | 1 | 18 ms |
| 4 KB | 400 | 405 | 3,791 ms | 7 | 70 ms |
| 211 KB | 100 | 102 | 3,207 ms | 1 | 37 ms |
| 211 KB | 400 | 405 | 12,803 ms | 7 | 193 ms |
| 833 KB | 100 | 102 | 9,251 ms | 1 | 93 ms |
| 833 KB | 400 | about 405 | more than 30 s (test deadline) | 7 | 567 ms
|

With 400 frames, the remaining saves are one per poll of up to 128
frames and one per controller acknowledgement.
- With the debug `paperclip-runnerd` and `fake-codex-app-server`
binaries from this branch: `npx vitest run` on
`src/live/runnerd-opencode-streaming-tools`,
`runnerd-opencode-assigned-tools`, `runnerd-codex-transport`,
`runnerd-codex-transport-settlement`,
`runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`,
`runnerd-saved-continuation`, `runnerd-detach-durability`, and
`src/cli/opencode-proxy-output`, `opencode-proxy-events`,
`opencode-app-server-proxy`: 11 files, 353 tests passed.

## Risks

- Durability. The rule is: nothing derived from a frame leaves runnerd
before the state that holds that frame is on disk. That covers delivered
events, command results, park decisions, tool results and rejections,
courtesy rejections, Cursor Cloud terminal acknowledgements, and trace
records that name event ids. The tests above check each of these.
- The unsaved window is at most one poll (128 frames). Every frame in it
is undelivered: no event from it reached the controller, and runnerd
wrote nothing to the provider while it was unsaved.
`a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`
proves this. A crash in the window therefore leaves the saved state and
everything the controller and provider have seen exactly as a crash just
before those frames were read. The resumed run starts from the same
event sequence, so it cannot deliver an event id twice. Its normal
recovery (thread resume and `thread/read` reconciliation) handles those
frames in the same way as frames still in the provider pipe. On
`master`, a crash could already lose the one frame being processed.
- One provider write can still happen in the window: `thread/read`,
which `CodexProvider::poll` sends to check the lineage of an unknown
helper thread. It is a read-only query that waits for its answer, and it
changes nothing on the provider side.
- Courtesy rejections now reach the provider at the end of the poll
instead of during it. The delay is one batch. The provider gets the same
replies.
- Previous-release story: the saved file shape does not change, and no
new field is saved. A file from the previous release loads as before.
The skip check compares exact bytes, so an older file is never treated
as already saved. The first save after an upgrade rewrites it in the
current shape. A deploy that lands mid-turn sees a file that the
previous release saved after each frame. That file is valid input for
this code.
- The ACPX backend has its own frame loop and is not changed here.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file editing) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:43:27 -06:00
DottaandClaude Opus 5.5 9c18cf61da fix(runner): keep discarded OpenCode token deltas from delaying tool calls (#20209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode runs use a TypeScript app-server proxy between
OpenCode and runnerd
> - runnerd reads the proxy's frames in order, only after the controller
acknowledges earlier events
> - For each frame, runnerd rewrites its whole durable provider state
with fsync, including every tool result that the turn retains
> - The proxy sent one `item/delta` frame for each reasoning or
tool-output token, and runnerd discards `item/delta`
> - A Paperclip tool call request therefore waited behind hundreds of
discarded frames, and the wait grew with each tool result in the turn
> - This pull request stops the proxy from sending the token deltas that
runnerd does not consume
> - The benefit is that tool calls reach runnerd without that queue, so
agent turns finish faster

## Linked Issues or Issue Description

**What happened**

In the chat eval `plugins.configure-plugin` (runner-opencode,
deepseek-v4.1-flash, GitHub Actions run 42574640038),
`paperclip_search_api` calls in one turn slowed from about 1.4 s to 9-15
s each. The server handled every call in 50-250 ms (`tool.execution`
spans). For one `call_api` call, 6.3 s passed between the server's 403
and the next agent activity. Time to first token was about 4.4 s.

The run events show the cause. Events that OpenCode streams reach
runnerd in bursts. In one burst, the usage events for three different
model requests arrived within 0.5 s, which no model can do. Each burst
ends with the next tool call request. 143 of 203 recorded
runner-opencode runs that we checked show such bursts, and 108 show a
gap of 8 s or more. Many chat eval cases that time out use this runner.

The mechanism:

1. OpenCode streams reasoning one token at a time. The proxy forwarded
each token as its own `item/delta` frame.
2. runnerd reads proxy frames in order and only after the controller
acknowledges earlier events. For each frame it calls `save_state`. That
serializes the whole provider state, with the tool results retained in
the turn, and fsyncs it.
3. runnerd's Codex facade has no mapping for `item/delta`, so it
discards the frame after the save.
4. A tool call request is one more frame in the same ordered stream. It
waits until runnerd works through every earlier delta frame. The wait is
about (reasoning tokens) x (save cost), and the save cost grows with
each retained tool result.

**Expected behavior**

A Paperclip tool call reaches runnerd soon after OpenCode makes it. Its
delay does not grow with the reasoning length or with the number of
earlier tool calls in the turn.

**Steps to reproduce**

1. Run a native OpenCode chat turn with a reasoning model that makes
many sequential Paperclip tool calls, for example `search_api`.
2. Compare each tool's `item.completed` tool_result time with the
`tool.execution.completed` time and with the next `item.started`
tool_use time in the run events.
3. See that the gap grows during the turn, and that events for several
model requests arrive within less than one second.
4. For a local model of the mechanism: run the pinned OpenCode binary
with a controlled OpenAI-compatible provider that streams 300 reasoning
tokens before each tool call. Pass the driver events through the proxy's
mapping and batching into a reader that consumes one frame at a time
with a save cost that grows by 4 ms per completed tool result. Before
this change, 8 sequential calls waited 1.7, 3.1, 4.2, 5.4, 6.7, 7.9, 9.2
and 10.5 s behind 142-261 queued frames. After this change, they waited
64-93 ms behind 0 frames.

## What Changed

- `shouldForwardOpenCodeProxyItem` now takes the event type and payload.
For `item.delta`, it forwards only assistant text (`agentMessage` or
`text`), which runnerd maps to `item/agentMessage/delta`.
- The proxy keeps reasoning and tool-output token deltas. Completed
reasoning and tool items still go to runnerd as `item/completed`. Tool
progress still goes as `item/updated`.
- New regression test: a step with 1,000 streamed reasoning and
tool-output tokens puts the same number of frames before the tool call
request as a step with 1 token, and no `item/delta` frame is written.
- New test: assistant text deltas are still forwarded. Completed
reasoning items are still forwarded.

## Verification

- `pnpm exec vitest run src/cli/opencode-proxy-events.test.ts
src/cli/opencode-proxy-output.test.ts` in `packages/paperclip-runner`:
15 passed.
- Red/green: with the predicate change reverted, the two new tests fail.
With the change, they pass.
- `tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`
has no errors in the changed files. Local errors exist only in
`src/eval/workflow-harness.ts`, which needs a built
`@paperclipai/paperclip-eval-kernel`.
- Local before/after measurement with the real pinned OpenCode binary
(`opencode-ai@1.18.34`), described in Steps to reproduce, step 4.
- Not run locally: the suites that need the `paperclip-runnerd` debug
binary or the bundled provider entrypoints
(`runnerd-opencode-streaming-tools`, `runnerd-codex-transport`,
`opencode-proxy-thread-read`). They fail the same way on the unchanged
base in this worktree because those build outputs are missing. CI covers
them.

## Risks

- Low risk. The changed frames were discarded by runnerd. No persisted
state shape changes, so no previous-release story is needed.
- If runnerd later maps a reasoning delta method, the proxy must forward
it with batching, as it does for assistant text. The test names this
boundary.
- This change does not change the per-frame durable save in runnerd.
Mapped frames (assistant text, tool items) still pay it. A runnerd
change that skips the save when a notification changes no state would
protect other proxies too. That is a separate change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, local benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:23:44 -06:00
DottaandClaude Opus 5.5 42bc238aa5 fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the run records it
as `provider_turn_lost_on_restore` and continues in a same-conversation
continuation turn
> - #20118 taught the event consumer to skip a stale restart-lost
terminal of another turn once a continuation is running. It applied only
when the checkpoint already recorded the interruption
> - When the checkpoint instead names the lost turn as active (the drain
hit right after the first turn was submitted), the consumer starts the
continuation on the first lost terminal. If the same lost terminal is
then delivered again, it was taken as the run's terminal and failed the
run
> - This pull request applies the skip to any recovered run once its
continuation has started. For a checkpointed interruption it still waits
for the startup decision first
> - The benefit is that a turn lost on restore always continues, however
often its lost terminal is delivered

## Linked Issues or Issue Description

Refs #20118

**What happened?**
Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master
9b71d03215): 94 of 95 passed. In t90 (claude M1 parent), run `e5eab3cb`:
1. The run submitted its first turn at 06:28:25, and a graceful drain
followed at 06:28:28.
2. After restore at 06:28:41, the ACPX runner replayed the turn's accept
and start, then reported `turn_lab_1abbeca2…` as
`provider_turn_lost_on_restore`.
3. The controller started continuation `turn_lab_a3d009c6…`; the saved
checkpoint names it as the active turn.
4. The run failed at 06:28:45 with `native_provider_terminal_failed: The
runner restored the conversation after process loss, but the previous
turn is no longer active.` and needed a Retry.

**Expected behavior**
Once the continuation is running, a later delivery of the lost turn's
terminal is history, not the run's terminal. The run finishes on the
continuation.

**Steps to reproduce**
1. Recover a run whose checkpoint names the interrupted turn as active
(not yet recorded as terminal).
2. Deliver that turn's restart-lost terminal, which starts the
continuation, then deliver it again before the continuation completes.
3. Before this change, the second delivery ends the run as failed.

**Paperclip version or commit**
9b71d03215 (wave). Branch rebased on current master.

## What Changed

- `native-session-runtime.ts`: `supersededByRestartContinuation` applies
to every recovered run.
- Once a continuation has started, a restart-lost `turn.failed` for any
turn other than that continuation keeps the consumer going.
- With a checkpointed interruption, it still waits for the startup
decision first. Before this change it excluded the checkpointed
interrupted turn itself, which could also end a run if that turn's
replayed terminal arrived before the startup continuation began.
- `native-session-runtime.test.ts`:
  - a lost terminal delivered again after the continuation starts;
- the checkpointed interrupted turn's own terminal arriving before the
continuation starts.

## Verification

- Red/green: both new cases fail on master (the run ends failed or on
the stale terminal) and pass with the change. The #20118 cases still
pass.
- `vitest run` over the runner restart, recovery, continuation and
terminal test files: 9 files, 389 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm run typecheck:typescript`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The skip still requires a recovered run, a recoverable
`provider_turn_lost_on_restore` failure, a continuation that actually
started, and a terminal for a different turn than that continuation. A
loss of the continuation itself, real provider failures, and runs that
started no continuation keep their current handling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:08:19 -06:00
Nicky LeachandClaude Opus 5.5 9046884811 perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner executes native agent runs. A host can park an idle
Runner (release it and freeze it) between runs, and resume it for the
next run on the same task.
> - A follow-up run that reuses a parked Runner must claim it first. The
claim is a bootstrap ticket, a checkpoint flush, and the plugin resume
that thaws the Runner.
> - Core made the claim only inside `attachRun`, after all of its
per-run preparation. The claim does not depend on that preparation, so
the two ran in series.
> - This pull request starts the claim while Core still prepares the
run. `attachRun` then waits for the claim that is already in progress.
> - The benefit is a shorter start for every follow-up run on a parked
Runner. The model below gives about 285 ms per run at P50.

## Linked Issues or Issue Description

No public issue exists. The issue is described inline.

**What happened?**

A follow-up run that reuses a parked Runner claims the Runner only in
`attachRun`. Core calls `attachRun` after backend setup: runner
environment, assigned MCP tools, tool authority and the plugin task
owner. The claim waits for all of this, but it uses none of it.

**Expected behavior**

The claim runs at the same time as backend setup. Everything that needs
the new run's tool authority stays at attach.

**Steps to reproduce**

1. Run a Codex or Claude task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. Before this change, the plugin resume starts only
after backend setup ends.

**Paperclip version or commit**

`master` at 7086019d4b.

**Deployment mode**

Hosted, with parked Runner tasks.

## What Changed

- **Transport** (`runnerd-codex-transport.ts`):
- New `beginRunAttachment({ runId })`. It runs the attach admission
checks, then starts the existing `#claimReleasedRunner` as the held
attachment operation. It returns `false` and does nothing when the
Runner is not released and confirmed, a claim already exists, or
admission fails.
- The admission checks move into `#assertRunAttachmentAdmission`.
`beginRunAttachment` runs them first, and `attachRun` runs them again.
- The next `attachRun` for the same run waits for the early claim. An
`attachRun` for a different run, `release` and `turn/start` still fail
with `native_runner_run_attachment_in_progress` while the claim is held.
- After an early claim, `attachRun` fails with
`native_runner_resume_reconciliation_expired` before it registers if the
claim deadline has passed.
- **Pass-throughs:** `HarnessNativeSession`, `CodexHarnessSession` and
the session, harness and transport contracts get an optional
`beginRunAttachment`. Each layer does the same identity,
protocol-integrity and busy checks that its attach does.
- **Core** (`native-session-executor.ts`):
- The executor records when the in-memory warm entry it reuses was
parked (`released`).
- `createRunnerdBackend` gets `beginRetainedRunAttachment`. Core gives
it only for that parked in-memory warm entry.
- The plugin task owner preparation moves into `preparePluginTaskOwner`.
On the early path, Core prepares and rebinds the retained owner at the
same time as the runner environment and the assigned MCP tools, then
starts the claim.
- Cold, adopted, remote warm recovery and restart paths keep the old
order.

### What stays at attach, and why

- Registration, the quiescence probe, `run.attach` and the authority
rotation stay in `attachRun`. Checkpoint registration needs the new
run's tool authority epoch, and Core creates that epoch during backend
setup.
- The admission checks run again at attach, so a state change during
preparation still blocks the attachment.

## Verification

- `vitest run src/live/runnerd-codex-transport-release-resume.test.ts`
(runner): 21 passed. Five new modes:
- `early claim`: `attachRun` waits for the held claim and does not route
the new run before the claim ends;
- `early claim abandoned`: a close after the claim ends fails the
waiting `attachRun` with `closed`, and the new run is never routed;
- `early claim cancellation`: a close during the claim cancels it, and
the next `attachRun` fails with `closed`;
- `early claim expired`: the deadline passes during the claim, and
attach fails before it registers;
- `early claim stale ticket`: the deadline passes after the claim,
before attach.
- `vitest run plugin-runner-release-resume.integration.test.ts
plugin-runner-eviction.integration.test.ts` (server, with real Runner
processes from `cargo build --workspace --bins`): 2 files, 51 passed,
none skipped. New modes:
- `cached held claim` (Codex and Claude): the plugin resume is held, and
the test proves Core's tool preparation runs while the claim is in
progress;
- `cached cancellation` (Codex): a cancellation after an early claim
retires the Runner and the run stops cleanly.
- `pnpm run typecheck:typescript` (runner) and `pnpm typecheck`
(server): pass.
- Evidence (modeled, not measured live): traces of 8 resumed runs on
parked Runners in a hosted test deployment, on current `master`. The
claim takes about 300 ms, and it fits inside backend setup in 7 of 8
runs. The modeled saving is about 285 ms per follow-up on a parked
Runner at P50 (range 208–331 ms). I did not run this change on a hosted
deployment.
- Not run: the full server suite (CI runs it), smoke tests, and the
hosted park, claim and attach loop with this change.

## Risks

- **New failure mode:** Core now claims the parked Runner before backend
setup ends. If that run then fails preparation, or is cancelled before
attach, the existing retire path closes the claimed session. The Runner
is retired, so that agent's next follow-up starts cold. Before this
change, a preparation failure happened before the claim, while the
Runner was still parked.
- **Ordering:** the retained plugin task owner is rebound to the new run
earlier, before Core creates tool authority. Nothing in the owner
preparation reads that authority. A failure in tool authority setup now
happens after the rebind, and the retire path handles it.
- **Scope:** only an in-memory warm entry that was parked takes the
early path. Cold launches, adopted Runners, remote warm recovery and
restart recovery are unchanged.
- **Deadline:** the claim deadline now starts earlier. A very slow
backend setup can make the claim expire before attach. Attach then fails
with `native_runner_resume_reconciliation_expired` before it registers,
the same failure as a slow claim today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, git, GitHub CLI) and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:08 -08:00
Nicky LeachandClaude Opus 5.5 92a18bb0f3 perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner drives Claude through the ACPX sidecar. The
sidecar's MCP endpoint lasts for the whole session, so it cannot tell
which turn sent a late tool callback.
> - Because of this, the Runner reaps the sidecar and provider of a
generation that ran a turn before it releases call-ID tombstones and
rotates receipt epochs. Today this restart runs at the start of the next
turn.
> - Every follow-up turn in a run pays that restart on its critical
path: about 1.5 s with real Claude. #20145 moves the restart to park,
but a second turn in the same run, after the first turn that follows an
attach, still restarts cold.
> - The restart does not have to wait for the next turn. When a turn
settles and the host has its terminal result, the Runner is idle.
> - This pull request restarts the provider at that point. The next
turn, or the next park, then finds a fresh generation.
> - The benefit: same-run follow-up turns start without a restart, park
does no restart after a settled turn, and the late-callback guarantee is
unchanged.

## Linked Issues or Issue Description

Refs #20145, which moved the restart to park. This pull request builds
on it. No public issue exists, so the problem is described here.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

`AcpxProviderSession::start_turn_timed` restarts the provider before any
turn whose generation already ran a turn. A follow-up turn in the same
run waits for `session.suspend`, sidecar shutdown, a new sidecar,
`initialize`, `session.open` and `run.attach` before `turn.start`. Since
#20145, park does the restart instead, but only for the first turn after
an attach.

**Proposed behavior**

When a turn settles and the controller has ACKed every committed event,
the Runner restarts the provider. The next `turn.start` uses the fresh
generation and only rotates receipts. Park reuses the fresh generation
and does no second restart.

**Reason and benefit**

The restart leaves the turn's critical path, both for same-run
follow-ups and for the turns after a park. The order of reap and
rotation stays the same, so a late callback from the reaped generation
still cannot reach a later turn.

## What Changed

- **Runner loop** (`durable/runner.rs`): new
`CommandExecutor::prepare_next_turn` hook, a no-op by default. The
runner calls it after the idle provider poll, only when no control frame
is waiting and the controller has ACKed every committed event
(`acked_source_seq >= highest_source_seq`). So the restart always runs
after the terminal result reaches the host.
- **ACPX turn-end restart** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- `restart_session_after_turn` restarts a Claude session that has a
completed turn, nothing pending, no buffered events, a dormant goal and
its whole event tail ACKed. It first settles an abandoned park
observation. If that settlement reads more events, it skips this round,
and the next idle round restarts after those events are ACKed.
- `AcpxProviderSession::restart_after_turn` does the same suspend → reap
→ start → bootstrap → identity check as the turn-start restart. It
records the reaped process id in `restarted_after_turn_from`. This field
is the explicit "fresh because of a turn-end restart" marker.
- The session does not poll a fresh generation until its next turn or
rebind, the same as a prepared one. Its startup output is published with
the next turn, as today. Polling it early would add events after the
host computed its park cut.
- **Next turn** (`acpx_provider_session.rs`): `start_turn_timed`
restarts only when the generation ran a turn. When
`restarted_after_turn_from` is set, it does the post-restart receipt and
ledger rotation without a second restart. `session.reconciled` reports
the reaped process as `previousProcessId`.
- **Park**: `prepare_next_generation` restarts (with the 20 s park
deadline) only if no turn-end restart ran. Otherwise it marks the
generation for `run.attach` rebind and returns `false`, so the runner
does not ask for a second park proof. The park-time receipt rotation
moves to the next turn start, where the turn-end path does it too.
- **Restart failure**: the executor sets the same state that a failed
turn-start restart leaves (`closed`, no session, provider exit
confirmed). The next `turn.start` fails with the same `ACPX turn start
failed closed: …` error that the turn-start restart gives today.
- **Fake sidecar**: new `pending-input` fault, which opens an input
request in a turn that then completes.

### Races

| Race | Handling |
| --- | --- |
| A turn or steer arrives during the turn-end restart | The runner loop
is single-threaded. The command waits in the socket until the restart
returns, then starts on the fresh generation without a restart. Only one
generation is live. |
| A turn arrives before the hook runs (for example, in the same read as
the ACK) | Control frames go first, so the turn restarts at its start,
as today. The hook then restarts after that turn. |
| Park arrives during the turn-end restart | Host control is read at the
top of the loop, so park waits. Park then reuses the fresh generation.
No event is published in between, so the host's park cut still matches.
|
| Park arrives before the hook runs | Park restarts with the 20 s
deadline, as in #20145. The hook then finds a fresh generation and does
nothing. |
| Eviction, `runner.suspend` or retire after the restart | They reap the
fresh generation through the ordinary paths. |
| The restart fails | The session is `closed`. The next turn fails with
the same error and the same durable state as a failed turn-start
restart. Park returns conflict, as it does for a closed session today. |
| Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after
the restart return `already_settled`, as after any settled turn. A
cancellation that retires the runtime closes the session, so no restart
runs. |
| Interrupted, cancelled or failed turn | `completed_turn_observed` is
false, so no turn-end restart runs. The next turn restarts at its start,
as today. |
| Pending permission or input requests at the terminal | The reducer
ends them at the terminal, so the restart runs. A later
`request.resolve` fails with `no active turn`, with or without the
restart. |
| Active session goal at the terminal | No restart, because the goal can
keep working after its prompt. |

### Removed as redundant, and what stays

- **Removed:** the park-time restart and dormant verify when a turn-end
restart already made the generation fresh. The runner's
`host_park_ready` proves dormancy just before `prepare_for_park`, and
nothing runs in between.
- **Removed:** the runner's second park proof in that case, because the
provider did not change.
- **Removed:** the park-time receipt and ledger rotation. The next turn
start does it once, for both paths.
- **Kept: the turn-start restart.** Without it, a turn queued before the
hook, a turn after an interrupted or cancelled turn, a turn after an
active goal, and any non-Claude ACPX turn would run in the generation
that ran the previous turn. Tests cover the queued-turn and interrupted
cases.
- **Kept: the park-time restart.** Without it, a park that arrives
before the hook would leave the next run to restart at attach. A test
covers this.
- **Kept: the settlement of an abandoned park observation.** Without it,
the restart drops events that the old generation sent before the
abandoned response. A test covers this, and it fails with the settlement
removed.
- **Kept: the goal-dormancy check.** Without it, the restart kills an
active goal's autonomous work. A test covers this, and it fails with the
check removed.
- **Kept: the ACK gate.** Without it, the restart can run before the
host has the terminal result. A runner test covers this.
- **Kept: deferred polling of the fresh generation.** Without it, its
startup event lands after the host computed its park cut, and the park
conflicts.
- **Kept: the kept failure message.** Without it, the next turn fails
with `ACPX session is not open`, and the restart error is lost.

## Verification

- `cargo test -p paperclip-runner-core --lib` and every
`tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery`
suite: all pass (debug).
- New tests (`tests/support/acpx_turn_end_restart.rs`, run from
`acpx_park_preparation`) prove these points:
- A settled turn reaps the old pid before it spawns the new one, and no
predecessor is still live. A same-run follow-up then sends only
`turn.start` and reports the reaped pid in `session.reconciled`.
- Park after a turn-end restart sends no sidecar command and returns
`false`. Attach rebinds the same pid. The run's second turn restarts at
its end, and the third turn starts without a restart.
  - The reaped generation's late callback never reaches the next turn.
- A turn that arrives before the hook restarts at its start. A park
before the hook restarts once.
- A failed turn-end restart gives the same next-turn error, retry error,
durable state and process count as a failed turn-start restart.
- Cancellation after the restart is `already_settled`. `runner.suspend`
reaps the fresh generation.
- An active goal skips the restart. An input request open at the
terminal behaves the same with or without the restart.
- `acpx_provider_turns`: a session-level turn-end restart drops the old
generation's late `issues.delete` callback, which reuses the previous
turn's call ID. The next turn then admits that call ID from the fresh
provider. Interrupted and retired turns do not restart at turn end.
- `acpx_park_readiness`: a turn-end-restarted generation is park-ready.
An abandoned park observation tail is delivered before the restart.
- `durable::runner`: the hook runs only after the terminal is ACKed and
when control is idle. It never runs during ACK debt, a waiting control
frame or a disconnect.
- Server integration tests (`plugin-runner-release-resume`,
`plugin-runner-eviction` and `plugin-runner-task-warm`) with this
branch's Runner: 49 passed. The provider process counts do not change:
each released run still adds one process start, now at turn end instead
of at park.
- `cargo fmt --check`: clean.
- Fake-sidecar timing (release build, 7 runs, medians):

  | | before (`master` with #20145) | this PR |
  | --- | --- | --- |
| same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms |
| second turn after an attach, `turn.start` → `turn.started` | 270.3 ms
| 23.5 ms |
  | turn-end restart, after the terminal is ACKed | — | 29.0 ms |
  | park preparation after a settled turn | 34.1 ms | 0.03 ms |

- Real Claude: not run for this pull request. With real Claude, the
restart took about 1.5 s on the turn-start path, and this change takes
it off that path. That number is not measured on this branch.

## Risks

- A run that ends after one turn without a park now starts one extra
sidecar that no turn uses. The Claude CLI itself starts only at the
first prompt, so the cost is the sidecar's start and bootstrap.
- The restart blocks the Runner loop for its length (about 1.5 s with
real Claude), as the turn-start restart does today. It has no deadline,
as before. A park that arrives during it waits, inside the host's 30 s
park wait.
- The fresh generation is not polled until its next turn. If it exits
while idle, the next turn finds out at `turn.start` and fails closed.
Park finds out at its observation.
- The turn-end restart is for Claude only, the same as park preparation.
Other ACPX agents keep restarting at turn start.
- If the Runner crashes during the restart, the durable state still says
`session_open`. Recovery is the same as for a crash while idle.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:34:26 -08:00
Nicky LeachandClaude Opus 5.5 2d93413085 perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path

> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.

## Linked Issues or Issue Description

No public issue exists. The issue is described inline.

### What happened?

On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.

### Expected behavior

The respawn happens while the Runner parks, so the next run starts on a
prepared generation.

### Steps to reproduce

1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.

### Paperclip version or commit

`master` at e194e193.

### Deployment mode

Hosted, with parked Runner tasks.

## What Changed

- **`durable/runner.rs`:**
- New `CommandExecutor::prepare_for_park(deadline)` hook, a no-op by
default. The runner passes a deadline 20 s after the park request; the
host waits 30 s for the reply.
- The Park host-control arm calls it after `host_park_ready()`,
re-proves readiness, and only then replies `parked`.
- A failure records a diagnostic and replies `conflict`, the existing
park-failure path.
- **ACPX, park side** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- For Claude, with the session open, nothing pending and nothing queued:
reap the old generation, using the same suspend-and-reap sequence a
second turn uses.
- Start the replacement with the current run attached, and prove it
dormant.
  - Roll the tool receipt epochs and the turn-identity ledger over.
- Record a digest of the session configuration it was opened with,
ignoring the MCP binding id and the aggregate digest.
- Every sidecar request during preparation is capped at the park
deadline. Running out of time fails like any other preparation failure.
- **ACPX, attach side:**
- A prepared, never-used generation whose digest matches the next run is
rebound: the sidecar gets `run.attach` with the new run id, the reducer
and tool bridges are reset, and the snapshot is checked for the new run
id.
- Anything else takes today's cold path: a different tool set, skill,
prompt or launch; a buffered callback; or any rebind error.
- **ACPX, turn start:** the provider restarts when its generation has
run a turn, instead of when the run has settled turns.
`has_settled_turns` is removed.
- **ACPX, polling:** polling skips a prepared generation until it is
rebound or starts a turn, so nothing it emits is attributed to the
parked run.
- **Sidecar** (`acpx-runtime-sidecar.ts`): rejects `run.attach` once a
turn has started in that process.
- **Fake sidecar:** reports live predecessors at process start, and adds
the faults `reject-open`, `late-tool-after-suspend` and `pre-turn-tool`.

## Verification

- `cargo test -p paperclip-runner-core`: 1029 passed (debug).
  - New `tests/acpx_park_preparation.rs` (10 tests). These prove:
- park reaps the old pid before spawning the new one, and acks only
after the dormant verify;
- a late callback from the old generation after park is rejected, and so
is a pre-turn call from the new generation;
- a compatible attach keeps the pid, and the sidecar and snapshot carry
the new run id;
- changed instructions, tools, skills or runtime context take the cold
path;
    - a used generation is never rebound;
- a prepare failure produces a park conflict and a recoverable state;
    - Retire works after a prepared park;
- a replacement whose `session.open` stalls fails at the park deadline,
leaves the session suspended, and the next run attaches and completes a
turn.
- `cargo fmt --check` and `cargo check --workspace --locked`: clean.
- `vitest` `acpx-runtime-sidecar.test.ts` and
`acpx-sidecar-input.test.ts`: 126 passed. The new re-attach guard test
fails if the guard is removed.
- `vitest` `plugin-runner-release-resume.integration.test.ts` and
`plugin-runner-eviction.integration.test.ts` with the branch's Runner:
48 passed. Both now expect the provider process to be replaced at park
instead of at attach.
- Fake-sidecar timing (release build, 3 runs): `run.attach` took 230–260
ms cold and 40–80 ms after a prepared park.
- Real Claude (claude-agent-acp 0.73.0, release build on a hosted test
VM, 3 runs each way plus one with a 180 s cgroup freeze between park and
attach):

  | | before | this PR |
  |---|---|---|
  | `run.attach` | 1691 ms mean | 212 ms mean |
| `run.attach` start → follow-up `turn.started` | 2565 ms mean | 1065 ms
mean |
  | `turn.start` → first provider output | 2261 ms mean | 2232 ms mean |
  | park preparation added to park | — | about 1450 ms |

Session recall, an authorized tool call after the rebind, and the
absence of any tool call or event from the old run all passed in every
run, including after the freeze. The prepared generation holds only the
sidecar: the Claude CLI starts at the follow-up prompt on both paths, so
a parked Runner also holds one node process instead of the whole Claude
tree.
- Hit rate: on a hosted test deployment, all 15 consecutive Claude runs
across two tasks carried identical content-addressed instruction and
skill bundle paths, so the prepared digest would have matched.
- Not run: the hosted park, claim and attach loop with real Claude, a
native MCP binding or skills with real Claude, `cargo test --release`,
smoke tests.
- Pre-existing flake:
`unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails intermittently on clean `master` when the `hosted acpx durable::`
subset runs in parallel. It passed in both full runs.

## Risks

- **Park takes about 1.5 s longer** before it replies with real Claude.
Preparation is capped at 20 s; the host's park wait is 30 s.
- **Configuration changes:** a change between park and the next run
(registered asset paths, tools, prompt) takes the cold path. That's
safe, but saves nothing.
- **Stale environment:** a prepared process was opened with the previous
run's environment and disk state. If credential or configuration files
change on disk while the Runner is parked, the prepared process won't
see them, whereas the cold path would.
- **Failed prepare:** a park that fails after the reap leaves no live
provider. The next turn or `run.attach` starts Claude cold, as today.
- **Scope:** only ACPX Claude prepares. Other providers park exactly as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, git, GitHub CLI) and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:38:57 -08:00
DottaandClaude Opus 5.5 b54317d3fb fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.

## Linked Issues or Issue Description

**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.

**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.

**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.

**Paperclip version or commit**
Staging revision b43cccbb0.

## What Changed

- `server/src/services/native-runtime/first-turn-queued-input.ts` (new):
`claimFirstTurnQueuedInput` seals a run's first-turn input in one
transaction. It locks the task, the deferred wakes, and the run in the
same order as the queue routes. It takes the responsible user's plain
queued comments in queue order (not interaction answers, interrupt
intents, independent continuations, conversations, or other users'
messages). It renders them with source-trust sanitizing and secret
redaction, then writes a `firstTurnQueuedInput` receipt
(`paperclip.first-turn-queued-input.v1`) into the run context. A second
call for the same run returns the same messages.
`runAcceptsFirstTurnQueuedInput` is true for a running native (or
preparing native) run that has no receipt and no accepted provider turn.
- `packages/paperclip-runner/src/native-session-runtime.ts`: new
`getFirstTurnQueuedInput` option. The runner calls it immediately before
the first `session.startTurn` and appends the result to `task.prompt`
(or to `events` for a resumed-session continuation envelope). It is not
called for the disposition-only recovery turn.
- `native-session-executor.ts`: passes the claim to the runner. A failed
seal commits nothing, so the executor logs it and starts the turn
without the messages. The messages stay queued.
- `queued-comment-delivery.ts`: a message in a run's receipt counts as
delivered only after the provider accepted that run's turn
(`turn.accepted` or the provider's `turn.started`), and only if the
message was not edited after the seal. Release promotion, Send, and
later first turns use this rule. `explicit-native-continuation.ts` uses
the same rule for stopped native runs.
- `routes/issues.ts` Interrupt: when the target run has not sealed its
first turn and has no accepted turn, the route does not record interrupt
intent and does not cancel the run. The response has
`executionWait.reason = "first_turn_pending"`. Other runs are
interrupted as before.
- Queue views (`routes/issues.ts` and the wake-queue adapter) hide
messages that the active run took into its first turn. Queue edits keep
those ids on the wake, so a run that fails before acceptance still
leaves them queued.
- `run-state.ts`: the issue-control run projection includes the receipt,
so the queue poll can hide taken messages.
- `doc/execution-semantics.md`: new section "Queued messages before a
native run's first turn".

## Verification

- Red before the fix, green after:
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "first
turn|already started"`: Interrupt on a preparing run and on a claimed
native run before `turn.start` does not stop the run, and the first turn
gets both messages in order. Before the fix, both cases cancelled the
run.
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime.test.ts -t "first-turn queued input"`: the
claim runs immediately before `turn.start`, and the first turn's prompt
ends with the queued messages. Before the fix, the claim was never
called.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/first-turn-queued-input.test.ts`: a run that
fails before `turn.start` leaves the message queued. The next run
delivers it, and after that no later run or copied retry context takes
it again. Before the delivery rule, the "exactly once" and "accepted
then failed" cases failed.
- Safety cases that pass before and after: Interrupt still cancels a run
whose provider accepted a turn (including a run saved by the previous
release with no receipt) and a run whose first-turn input is already
sealed. The existing "explicit interrupt fallback" cases for preparing
and native runs without a responsible user still pass.
- Speed: the seal is one short transaction before `turn.start`. In the
embedded-Postgres test it took 5.5 ms. It removes a full second
preparation (54 s on staging) from the message-to-first-token time.
- Related suites: `src/modules/wake-queue`,
`explicit-native-continuation`, `issue-queued-comment-queue`,
`issue-queued-comments-routes`, `queued-comment-consumption`,
`heartbeat/run-state` (888 tests), `native-session-executor.test.ts`
(1306 tests), and the full `native-session-runtime.test.ts` (238 tests)
pass. `pnpm --filter @paperclipai/server typecheck` passes.
- Not done: no live run against a real provider or sandbox. Staging was
read only.

## Risks

- Previous-release state: a deferred wake saved by the previous release
has the same shape and is taken normally. A run saved by the previous
release has no receipt. If it is resumed before its first turn, it seals
on the new release. If its provider already accepted a turn, Interrupt
stops it as before. A run saved by the previous release while it was in
`turn.start` but before acceptance counts as unsealed, so in that short
window an Interrupt does not stop it and the message is delivered at the
turn boundary.
- Rollback: if the server rolls back after a run sealed and its turn was
accepted, the old release does not read the receipt and can deliver
those messages a second time.
- A first turn that does not use the fresh-turn path (restart
continuation, disposition-only recovery) does not call the claim. Its
messages are delivered at the turn boundary as before.
- A message that arrives after the seal waits for the turn boundary.
Interrupt stops the run as before.
- No schema migration. The receipt is a key in the run context.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution, read-only
browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:44:18 -06:00
DottaandClaude Opus 5.5 cedf335c98 fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the run records
that turn as failed with `provider_turn_lost_on_restore` and continues
on a same-conversation continuation turn
> - Restart recovery blocks a run as
`provider_checkpoint_permanently_failed` when its checkpoint carries a
failed run terminal and no pending restart interruption
> - While a continuation is running, the checkpoint keeps the lost
turn's failed run terminal and names the continuation as the active
turn. The pending-interruption check returns nothing once a turn is
active
> - So a second restart during the continuation read the run as
permanently failed, blocked replay, and needed the board
> - This pull request treats that failed terminal as the restart loss it
records whenever an active turn follows a restart-lost terminal turn
> - The benefit is that runs survive repeated restarts during their
continuations, including M1 parents and their subtasks

## Linked Issues or Issue Description

Refs #20148

**What happened?**
Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master
8e3b64a257), t05 (claude M1 parent):
1. Run `e1e37dab` lost turns to restarts at 00:41:32 and 00:47:53. Each
time it continued, and the third continuation `turn_lab_92b03dc7…` was
active.
2. At 00:48:29 the next restart blocked it:
`native_restart_recovery_blocked` /
`provider_checkpoint_permanently_failed`, with replay blocked.
3. Its saved checkpoint held `terminal.runTerminalState: "failed"` (from
the lost turn), `activeTurnId` set to the continuation, and a last
terminal turn fingerprinted `provider_turn_lost_on_restore`.

Its codex/ACPX subtask's run `f9b10d06` was blocked in the same pass
with the same shape, and its continuation had already proposed a
semantic result. Both stayed blocked. The parent's Retry was held on the
blocked child.

**Expected behavior**
A failed run terminal that records a restart loss, followed by an active
continuation turn, is not a permanent failure. Recovery claims
`resume_dead_runner`. The runtime then reconciles the active turn, and a
held result settles a lost turn as completed. A real provider failure
followed by a later turn still blocks.

**Steps to reproduce**
1. Checkpoint a run whose last terminal turn is a restart loss, whose
`terminal` is failed, and whose `activeTurnId` is a later continuation
(optionally with a semantic result).
2. Claim restart recovery. Before this change it blocks with
`provider_checkpoint_permanently_failed`.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `contracts/native-session-backend.ts` (runner): the new
`nativeRestartContinuationInFlight` is true when all of these hold:
  - the checkpoint has an active turn;
  - its last terminal turn is a different turn;
- that turn's fingerprint is a failed, result-less, recoverable
`provider_turn_lost_on_restore`.
- `native-session-runtime.ts`: recovery clears that stale failed
terminal for a continuation in flight, as it already does for a
checkpointed interruption. Before, it threw
`provider_checkpoint_failed_terminal` before it reconciled the active
turn.
- `native-restart-recovery.ts` (server): `checkpointFailed` now also
requires that no restart continuation be in flight. It uses the same
contract function through the vendor boundary.
- `native-session-runtime.test.ts`: an end-to-end recovery test. A
checkpoint with the stale failed terminal and an active continuation
resolves on the continuation's result without starting a new turn.
- `native-runner-restart-recovery.integration.test.ts`:
  - a codex case of the stale-terminal shape as saved in production;
  - an ACPX case with a held result;
- a held case where an active turn follows a real provider failure
(`model_not_found`).

## Verification

- Red/green: on master, the stale-terminal case is blocked with
`provider_checkpoint_permanently_failed`. With the change it claims
`resume_dead_runner`. The real-failure case stays blocked on both.
- Red/green (runtime): without the runtime change, the end-to-end test
throws `provider_checkpoint_failed_terminal`. With it, the run
completes.
- `vitest run` over the restart-recovery and failed-workspace-recovery
test files in `server/src/services/native-runtime`: 5 files, 305 tests
passed (14 skipped).
- `vitest run` over the runner restart, recovery, continuation and
terminal test files: 9 files, 385 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`) and `pnpm
run typecheck:typescript` (`packages/paperclip-runner`): passed.

## Risks

- Low risk. The exception applies only when the recorded failure is
itself a recoverable restart loss and a later turn is in flight. Every
other recovery gate is unchanged: the provider-attempt budget,
controller takeover and process proofs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:28:12 -06:00
DottaandClaude Opus 5.5 b43cccbb08 fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the runner reports
it as `provider_turn_lost_on_restore` and the controller starts one
same-conversation continuation. A later restart that loses the
continuation is answered the same way
> - With back-to-back restarts, the controller's checkpoint can miss a
continuation that the runner already accepted. On the next restore the
controller starts a fresh continuation, and the restored runner then
reports the unrecorded earlier continuation as lost
> - The event consumer treated that stale lost terminal as the run's
terminal, so the run failed while its new continuation was already
running
> - Recovery then refused to replay the failed run, and the task needed
a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but
the parent stayed `blocked` on that subtask
> - This pull request makes the consumer skip a restart-lost terminal
for any turn other than the continuation it started
> - The benefit is that back-to-back restarts no longer turn a
recoverable lost turn into a failed run that needs Retry

## Linked Issues or Issue Description

Refs #20107

**What happened?**
Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade
between builds plus repeated restarts) at 070365404e. In task t02
(claude M1), the codex subtask CHA-7 failed with
`native_provider_terminal_failed: The runner restored the conversation
after process loss, but the previous turn is no longer active.` The
sequence:
1. Restore 1 lost turn `01a1bd17…`, and the controller started
continuation `01a1bd18-15ec…` (runner command 14). A restart followed
within a second, before the controller's checkpoint recorded that
continuation.
2. Restore 2 saw the checkpointed interruption of `01a1bd17…` and
started a fresh continuation, `01a1bd18-4cc4…` (command 24). The
restored runner also reported the unrecorded continuation
`01a1bd18-15ec…` as `provider_turn_lost_on_restore`.
3. Because a continuation was already started, the consumer returned
that stale terminal as the run's terminal. The run failed, the new
continuation was stopped (command 25), and automatic recovery refused to
replay it (`uncertain_provider_action`, `preserve_without_replay_v1`).
4. The parent CHA-2 asked CHA-7 to resume by comment, but an agent
comment cannot authorize a blocked replay, so the wake stayed
`deferred_issue_execution`. The parent's own Retry succeeded, but CHA-2
stayed `blocked` on CHA-7 (`R3:retry_did_not_complete`).

**Expected behavior**
A restart-lost terminal for an older, unrecorded continuation is
superseded by the continuation this restore started. The run continues
on that continuation and needs no Retry.

**Steps to reproduce**
1. Checkpoint a native session whose last terminal is a restart-lost
turn and whose active turn is null, so recovery starts a continuation.
2. Have the restored session report another turn (the unrecorded earlier
continuation) as `provider_turn_lost_on_restore`, either before or after
the new continuation starts.
3. Before this change, the run ends on that stale terminal.

**Paperclip version or commit**
070365404e (wave); branch rebased on current master.

## What Changed

- `native-session-runtime.ts`:
  - Records the turn id of the restart continuation it starts.
- New `supersededByRestartContinuation`, used by the consumer's
`resumeInterruptedTurn`, keeps consuming past a
`provider_turn_lost_on_restore` `turn.failed` when all of these hold:
    - the run was recovered from a checkpointed interruption;
- the failed turn is neither the checkpointed interrupted turn nor the
started continuation;
    - a continuation was actually started.
- The check first waits for the startup decision, because the consumer
starts before the continuation does.
- `native-session-runtime.test.ts`: a red/green test, with the stale
lost terminal arriving both after and before the continuation starts.

## Verification

- Red/green:
  - On master, both test cases end the run on the stale terminal.
- With the change, both resolve on the continuation's result. The
continuation is started once, and the stale `turn.failed` is still
appended to the journal.
- `vitest run` over the restart, recovery, continuation and terminal
test files in `packages/paperclip-runner`, including all of
`native-session-runtime.test.ts`: 9 files, 380 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck:typescript`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The skip applies only to restart-lost failures of turns
other than the checkpointed interrupted turn and the started
continuation, and only after a continuation actually started. Real
provider failures, a loss of the continuation itself, and runs without a
checkpointed interruption keep their current handling.
- If startup fails, the startup decision still settles. The consumer
then falls back to the existing behavior and the run fails as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:44:13 -06:00
DottaandClaude Opus 5.5 412d49e48d fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path

- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.

## Linked Issues or Issue Description

Refs #18879

**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.

**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.

**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.

## What Changed

-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.

## Verification

- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at 070365404e, still failed for 3 Claude
runs. The cause was the install path: the saved sidecar path pointed
into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun
has been requested.

## Risks

- **The restored conversation runs on the new provider pack.** That is
already the case for re-bound suspended sessions. Session start still
verifies every artifact and the restored provider identity against the
qualified command digest. A changed upstream release still fails strict.
- **Old provider process.** For an active turn, lost-turn recovery waits
for the provider-lifetime fence before restoring. For an open session,
behavior matches the same-build restart, which reopens without the
fence.
- **A state that really is incompatible** (an upstream provider release
change):
- **Between turns**, the common upgrade case: the next run's checkpoint
carries `allow_replacement_after_resume_failure`, so the refusal opens a
fresh provider session automatically, with no Retry. A new test,
`continues in a fresh session when a new run's saved session is
incompatible…`, pins this in `native-session-runtime.test.ts`.
- **Mid-turn, same run:** this still stops for a board Retry, by design.
The effects of the lost turn are unknown, and the automatic replacement
sweep (`decideNativeReplacement`) only replaces a session when it has a
complete effect inventory.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:34:23 -06:00
DottaandClaude Opus 5.5 15b987d9b8 fix(runner): a finish or block waits for held calls before it is judged (#20108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Restarts must be invisible to a task: an agent that was working
keeps working after the control plane comes back
> - Restart chaos t05 (Daytona warm wave, codex, K4 hold) ended a task
as blocked even though its document save succeeded
> - #20069 refuses a finish or block that the agent decided before it
saw results that had already returned. Here the results had not returned
yet when the block was checked: after the reattach, the held calls and
the later block are handled at the same time
> - This pull request makes a finish or block wait, for a bounded time,
for earlier calls of its turn that have not returned, and then apply the
same check
> - The benefit is that a controller outage during a Paperclip tool call
no longer turns into a false "blocked" task

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261108T192459-s9501`, task t05 (Daytona warm, codex, C1, K4 hold of
182 s, master c5a172a183, which includes #20069).

**Problem**

1. 19:37:01: the follow-up comment arrives. The agent reads the brief
and calls `write_document`.
2. 19:37:44 to 19:40:51: K4 hold. Codex keeps the call pending and
re-samples the model: "The save request is still pending", then
`read_document` (also pending), then `paperclip_block`.
3. 19:42:02, after the reattach: the controller recovers the held write
and read (`recoverPendingSemanticInputs`) while the Runner delivers the
block. The block reaches the check at seq ~331–334. The write's result,
with the FOLLOWUP line saved, is committed at seq 337. The check saw no
returned call, so the block was recorded and the issue rested `blocked`
with no Retry.

So held calls are answered after ready (the save succeeded, the brief
has the marker). The defect was the order in which the terminal call was
judged.

## What Changed

- `packages/paperclip-runner/src/live/terminal-call-freshness.ts`:
- `pendingEarlierToolCalls`: calls made before the terminal call that
have neither returned nor reached the agent;
- `settleEarlierToolCalls`: waits, bounded at 30 s
(`TERMINAL_CALL_SETTLE_TIMEOUT_MS`), until none is pending, then reports
which returned unseen and which are still pending;
- `staleTerminalCallResult` also names still-pending calls: "has not
returned yet. Wait for its result (do not repeat a write until it
returns)…", still `terminal_call_before_results`, retryable.
- `runnerd-codex-transport.ts`: before dispatching `paperclip_finish` or
`paperclip_block`, waits for those calls and re-checks session identity.
It refuses the terminal call if any earlier call returned unseen or is
still pending.
- Scope: `DurablePrpCodexTransport` serves Codex and ACPX on local, warm
Daytona and per-turn Daytona, so every path that reattaches to a
surviving Runner is covered. The wait applies only to terminal calls
with earlier unreturned calls, so it adds nothing before `turn.start`.
- Not covered: the external-driver path
(`drivers/external/runnerd-external-driver.ts`) has its own semantic
handler without either check. It is recorded as a campaign follow-up.

## Verification

Red/green:
- `runnerd-codex-transport.test.ts` "refuses a terminal call that
arrives before a held call's result (restart chaos t05)": the write is
started without a result, the block arrives, and the write's
`semantic_tool.result` lands 100 ms later.
- On master's transport and guard: fails, the block is dispatched
(`handler` called once).
- With this PR: the block is refused with
`terminal_call_before_results`, naming the returned write.
- `terminal-call-freshness.test.ts`: pending versus returned, a result
that lands during the wait, a call still pending after the bound, no
wait when everything was seen, and a session that goes away.
- Both files: 300 passed. `tsc --noEmit -p packages/paperclip-runner` is
clean.

Live: the crash point `runner.semantic_tool.after_input_before_dispatch`
(from #20069) holds K4 during a Paperclip tool call. A Daytona run
against this branch is requested from the slot owner after merge.

## Risks

- A finish or block made in the same model step as a write (parallel
tool calls) now waits for the write and is refused once, because the
agent had not seen the write's result. The agent calls it again after
reading the result: one extra model step in that case. #20069 already
behaved this way whenever the write returned first.
- A Paperclip call that never returns delays a terminal call by 30 s,
and the terminal call is refused. Recovery answers held calls after
ready or disconnects the Runner, so the refusal cannot repeat
indefinitely in practice.
- No schema or persisted run-state change.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (code
comments; no user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:08:06 -06:00
DottaandClaude Opus 5.5 a8bbdf237c fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry

## Linked Issues or Issue Description

Refs #20089

**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.

**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.

**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".

**Paperclip version or commit**
master at f789f3e2af. Branch rebased on current master.

## What Changed

- `acpx_provider_backend.rs`: `turn.start` checks whether there is no
live session and the saved state is reopenable. If so, it reopens
through `open_session` first, and the reopen's `session.resumed` event
comes before the turn's events. The previous body moved unchanged into
`start_turn_in_open_session`.
- `tests/acpx_warm_attachment.rs`: a new test,
`reopens_the_conversation_for_a_turn_after_a_suspension_stop`. It drives
prepare, open, turn, stop, turn and expects the second turn to be
accepted with `session.resumed`. It also checks that after
`session.close`, `turn.start` still fails.

## Verification

- Red/green: the new test fails on master with "ACPX session is not
open" and passes with the change. `cargo test -p runner-core --test
acpx_warm_attachment`: 3/3 passed.
- I also ran the full `runner-core` crate suite. Five tests failed: four
`task_memory` monitor tests and `unconfirmed_suspension_state...`. Each
of them passes on an unchanged master checkout, and the ACPX one also
passes when rerun in this branch, so they are timing-dependent and not
caused by this change.

## Risks

- Low risk. A reopen happens only under the same conditions in which
`session.open` would already succeed for this run. A closed session and
an unconfirmed provider exit stay fail-closed. The order of events
changes only by adding `session.resumed` ahead of the turn's events.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:46:11 -06:00
DottaandClaude Opus 5.5 a88e23fa34 fix(runner): read an uncommitted turn start as empty history when the event window is complete (#20107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run must survive repeated control-plane restarts; each
restore reconciles the turn its checkpoint names as active
> - That reconciliation reads the turn's history. When the provider has
no paginated history, the runnerd transport rebuilds it from the
committed runner events
> - After back-to-back hard restarts, the checkpoint can name an
accepted turn whose start was never committed. The runner was killed
after journaling `turn.start` and before the provider emitted anything
> - The rebuild treated every missing start as lost history and failed
with `codex_history_incomplete`, so the task needed a Retry
> - This pull request treats a missing start as empty history when the
committed window is provably complete
> - The benefit is that a run survives back-to-back restores, as it
already survives one

## Linked Issues or Issue Description

Refs #20074

**What happened?**
In restart-chaos wave w-20261108T200620-s9701 (local, K4/K6 + M1,
back-to-back K4 restores), three runs failed on the second restore. Two
were Claude ACPX C1 tasks; the third was an M1 parent with its subtask.
They failed with `native_session_recovery_failed:
HarnessReconciliationError: codex_history_read_failed: thread/items/list
requires supported paginated history; Error: codex_history_incomplete:
requested turn start is outside the retained runner event window`, and
each needed a Retry.

The sequence:
1. The first restore lost the in-flight turn and started a continuation
turn. ACPX echoes the requested turn id, so the checkpoint named it as
active.
2. Six seconds later a second K4 SIGKILL killed the runner. The runner
had journaled that `turn.start` (now `execution_indeterminate`), but the
provider had emitted nothing, so no `turn.started` was committed and the
runner outbox was empty.
3. On the second restore, `thread/items/list` for that turn hit the
guard, although the committed window was complete: source sequences
1–25, nothing evicted.

**Expected behavior**
The second restore treats the never-started turn like the first restore
treated its lost turn. The runner reports it lost on restore, and
recovery starts a continuation with no Retry.

**Steps to reproduce**
1. Have a checkpoint name an accepted turn whose `turn.started` was
never committed. For example, kill the runner after `turn.start` and
before the provider emits anything.
2. Restore the session. Before this change, recovery fails with
`codex_history_incomplete`.

**Paperclip version or commit**
master at 88835e624a plus #20094 (the wave). Branch rebased on current
master.

## What Changed

- `runnerd-codex-transport.ts`: a new
`committedEventWindowComplete(state)`. It is true only when the
committed events are exactly source sequences 1 to `ackedSourceSeq`:
none missing, none duplicated, none evicted.
- In the `thread/items/list` fallback, a turn with no committed start
returns empty history when the window is complete. Otherwise it still
throws `codex_history_incomplete`.

## Verification

- Red/green connected test in `runnerd-restart-continuation.test.ts`:
"starts a new turn when the checkpoint names an accepted turn whose
start was never committed". It uses the real runnerd and the fake Codex
server, and holds the provider start with `--turn-started-gate`.
  - Without the change, it fails with the exact production error.
  - With the change, the run completes on a continuation turn.
- Unit test: `committedEventWindowComplete` is false for a gap, a
missing first event, duplicates and no state.
- `npx vitest run` over every Codex, restart and recovery test file in
`packages/paperclip-runner`: 45 files, 1158 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The empty history applies only when every acknowledged event
is still retained, so no event of that turn can have existed and been
dropped. Any evicted or gapped window keeps failing closed.
- Known gap, recorded as a follow-up: if eviction did remove events of a
turn the checkpoint still names, recovery still fails closed. The
planned fix is to never evict events of an unresolved checkpointed turn.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:37:11 -06:00
DottaandClaude Opus 5.5 a9ae1b69c3 fix(runner): let a finish through after a restart lost the provider turn (#20088)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner relays an agent's Paperclip tool calls and keeps them
across control-plane restarts
> - #20069 refuses a `paperclip_finish` or `paperclip_block` decided
before an earlier returned result reached the agent
> - The guard filters by turn id, but each committed event carries the
run's one identity turn id, so provider turns are not separated
> - When a restart loses a provider turn, a call of that turn gets its
result but never its provider completion
> - The guard then refuses every later finish of the run, so the run
exhausts its retries and the user must click Retry
> - This pull request makes a new provider turn end the pending calls of
the previous turn
> - The benefit is that an agent can finish after a restart that lost
its turn

## Linked Issues or Issue Description

Refs #20069

**What happened?**
A restart chaos storm (local, Codex, A1/C1, K2 drains for 30 minutes,
116 tasks) had one failure. Task t110 needed a user Retry. Its run was
drained twice. The first drain lost the provider turn while a
`read_agent_instructions` call was pending. After that, all four
`paperclip_finish` calls failed with `terminal_call_before_results` and
named `read_agent_instructions`. The run ended with
`native_session_retry_exhausted`.

**Steps to reproduce**
1. Start a Codex run that calls a Paperclip tool.
2. Drain the control plane while that call is pending, so the Runner
restores the session and reports `provider_turn_lost_on_restore`.
3. Let the agent continue in the new turn and call `paperclip_finish`.

**Expected behavior**
The finish is recorded. The lost call belongs to a turn that the agent
will not see again.

**Actual behavior**
The Runner refuses every finish with `terminal_call_before_results`. The
run fails and needs a user Retry.

## What Changed

- `unseenReturnedToolCalls` forgets the calls it tracked when a
`turn.started` event begins a new provider turn.
- A new test reproduces the t110 event order: a call in a lost turn,
then a new turn that reads, sees its result, and finishes.

## Verification

- `npx vitest run src/live/terminal-call-freshness.test.ts` in
`packages/paperclip-runner`: the new test fails before the change and
passes after it (7/7).
- `npx vitest run src/live/runnerd-codex-transport.test.ts -t terminal`:
10 passed.
- `tsc --noEmit -p packages/paperclip-runner`: clean.
- Evidence: the kept Runner control-plane state of t110 shows the call
`exec-0f3d1b90` (generation 1) with a `semantic_tool.result` command but
no `tool.execution.completed`, followed by `turn.failed
provider_turn_lost_on_restore` and new `turn.started` events.

## Risks

- The t03 case of #20069 stays covered: the stale block and the earlier
calls are in the same provider turn, with no `turn.started` between
them. The existing t03 tests still pass.
- If a driver emits `turn.started` inside one provider turn, the guard
can miss a stale call of that turn. No current driver does that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 12:29:38 -06:00
DottaandClaude Opus 5.5 c55af26821 fix(runner): treat a replayed start of a turn lost on restore as history (t48) (#20074)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run must survive a control-plane restart and continue
its work
> - #20057 made recovery adopt an in-flight turn, or start a new turn
when the restored runner reports the old turn lost
> - One case was not covered. A graceful drain can detach the controller
after the runner accepted `turn.start` but before the provider started
the turn
> - The runner then records the provider's `turn.started` while no
controller is attached. After a restart it replays that start, and the
Codex session rejects it as an unexpected turn
> - This pull request treats that replayed start as history from the
earlier generation
> - The benefit is that the run starts its continuation turn instead of
stopping with `native_event_replay_conflict`

## Linked Issues or Issue Description

Refs #20057

**What happened?**
A restart chaos wave ended with one failed task (t48, Codex). A graceful
SIGTERM suspended the controller 300 ms after the runner accepted
`turn.start`. The provider's `turn.started` came after the controller
detached. Recovery classified the run as `resume_dead_runner`. The
restored runner reported the turn as `provider_turn_lost_on_restore`,
which is correct. But it also replayed the undelivered `turn.started`.
The Codex session failed with "Provider started an unexpected turn."
(`turn_binding_mismatch`), and recovery stopped with
`native_event_replay_conflict`.

**Expected behavior**
Recovery starts a new continuation turn. The replayed start of the lost
turn is recorded as history, and the run completes.

**Steps to reproduce**
1. Start a Codex native run.
2. Detach the controller gracefully after the runner accepts
`turn.start` and before the provider emits `turn/started`.
3. Let the runner record `turn.started` while it is unattached, then
kill the runner and the provider.
4. Restore the run. Before this change, the session fails with
`turn_binding_mismatch`.

**Paperclip version or commit**
master at a9cf4ee731 (the wave ran on 66a2cbab98).

## What Changed

- `codex-app-server-driver-impl.ts`: recovery identifies the turn that
this run's latest `turn.start` had the runner accept, when recovery did
not adopt it and provider history does not show it live. Recovery passes
it to the session as `lostAcceptedTurnId`.
- `codex-session-state.ts`: the session keeps `lostAcceptedTurnId` as an
unconfirmed lost turn. The stale-event rule treats its replayed start
and items as history.
- `codex-session-notifications.ts`: `turn/started` applies the
stale-event rule to earlier-generation and unconfirmed lost turns. The
runner's restart-loss terminal confirms the lost turn. Any other
terminal, or a semantic result, from that turn fails closed, so a real
outcome is never dropped. A start for a turn that this generation
settled, or for a turn the session never saw, still fails closed.
- `runnerd-restart-continuation.test.ts`: a new connected test with the
real runner and the fake Codex server reproduces the t48 shape. A new
fake option, `--turn-started-gate`, holds the provider's start until the
test detaches the controller, so the test does not depend on timing.
- `codex-app-server-driver.terminal-replay.test.ts`: driver tests for
the replayed start (history), and for an ordinary failure or a result
from the lost turn (fail closed).

## Verification

- Red: before the fix, the new test failed with
`NativeProviderTerminalFailure: Provider started an unexpected turn.` at
`codex-harness-session.ts:211`. This is the same error and stack as the
chaos run.
- Green: `npx vitest run src/drivers/codex
src/live/runnerd-restart-continuation.test.ts` (in
`packages/paperclip-runner`) passed 392 of 392.
- The wider Codex, restart and recovery test files (45 files, 1153
tests) passed on the final revision. The two fail-closed driver tests
fail on the first revision, where the lost turn's outcome was dropped as
history. The one failure in an earlier run ("still rejects history for a
turn that became terminal in the live session") showed that the first
version was too broad. The rule now applies only to earlier-generation
turns, and that test passes.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in
`packages/paperclip-runner` passed.

## Risks

- Low risk. The change only applies after a restore, to a turn that the
runner accepted for this run's latest `turn.start`, and only when
recovery did not adopt that turn and provider history does not show it
in progress.
- A turn that the session never saw still fails closed. A re-start of a
turn this generation settled still fails closed. A real failure or a
result from the lost turn fails closed instead of being dropped.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:14:24 -06:00
DottaandClaude Opus 5.5 f679cf81dc fix(runner): refuse a finish or block decided before earlier results reached the agent (#20069)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Restarts must be invisible to a task: an agent that was working
keeps working after the control plane comes back
> - Restart chaos t03 (Daytona warm wave, K4 holds) ended a task as
blocked even though its document was saved
> - The cause is in how a terminal decision (paperclip_block or
paperclip_finish) is handled after a long controller outage, not in how
the Runner journals tool calls
> - This pull request refuses a terminal call that the agent made before
it had seen results that already returned, so the agent continues with
those results
> - The benefit is that a controller outage no longer turns into a false
"blocked" task

## Linked Issues or Issue Description

Restart chaos t03, wave `w-20261108T142906-s9501` (Daytona, codex, A1,
three back-to-back K4 holds of 136 s, 252 s and 152 s). Refs #19831
(durable ownership plan).

**What happened**

The kept instance's run log shows the order of events:
1. The turn is accepted at 14:29:57. The runner then delivers nothing to
the controller until 14:39:27. The holds keep the controller down, and
the Runner keeps Codex's `item/tool/call` requests until it is
reattached.
2. During that outage Codex keeps running. Its `write_document` call
stays pending. The model is sampled again about once a minute and says
"The save request is also waiting on Paperclip". It then calls
`paperclip_block`, with the reason `tool_unavailable`.
3. After the reattach the Runner delivers the backlog in order.
`write_document` runs and returns (the document is created: sourceSeq
152). Then the stale block arrives (sourceSeq 247) before Codex has
acknowledged the write result: there is no `tool.execution.completed`
for it before the block. The task ends blocked.

**Codex behavior, checked directly** against the pinned Codex 0.160
app-server. A local probe sent one dynamic tool call and left it
unanswered:
- Codex does not time out. It re-samples the model at about 60 s and 127
s ("The save call is still waiting to return").
- A reply sent 130 s later is accepted (`durationMs` 130003), and the
model finishes ("DONE").

So the reply is not lost. The agent's terminal decision was made without
it.

**Expected behavior**
- A terminal call decided while earlier Paperclip results were still
unseen is refused with a retryable error.
- The agent continues with those results. In t03 it would see that the
document was saved, and finish.

## What Changed

- `packages/paperclip-runner/src/live/terminal-call-freshness.ts`:
`unseenReturnedToolCalls` reads the turn's committed run events up to
the terminal call. It returns each dynamic call that has a returned
`tool_result` but no provider `tool.execution.completed`.
`staleTerminalCallResult` builds the retryable
`terminal_call_before_results` error, which names those calls.
- `runnerd-codex-transport.ts` `#handleSemanticToolInput`: when such
calls exist, `paperclip_finish` and `paperclip_block` get that error
instead of being dispatched. The recovery path after a restart uses the
same handler. A long-running call with no result yet does not count.
- Crash point `runner.semantic_tool.after_input_before_dispatch`, just
before the controller dispatches a dynamic tool call. A crash-point hold
(`--hold-ms`, `--crash-point-hold-ms`; a K4-style receipt) keeps the
controller down after the self-kill. Together they reproduce t03 on
Daytona.

## Verification

- Red/green: `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport.test.ts -t "refuses a terminal call
decided"`.
- The case `seen: false` failed on master: the handler was called and
the stale block was dispatched. It passes with the fix.
- The case `seen: true` (the agent saw the result first) dispatches
normally, both before and after.
- `npx vitest run src/live/runnerd-codex-transport.test.ts`: 289 passed.
`src/live/terminal-call-freshness.test.ts`: 5 passed.
- `npx tsc --noEmit -p packages/paperclip-runner` passes, and `node
scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- Shared registry: 15 passed. Call-site test: 3 passed. Chaos runner
tests: 12 passed.
- Codex probe (local, about 2.5 min per run): reproduced the "still
waiting" re-sampling and the accepted late reply described above.
- Not yet run: the Daytona crash-point repro with a hold. It needs the
shared harness slot, and I'll add the result here. The fix is controller
TypeScript, which is not an input of the Daytona image, so the current
image still matches.

## Risks

- A refused terminal call costs the agent one more turn step. It applies
only when a result returned that the agent had not seen, which in
practice happens only after a controller outage.
- The check reads committed run events. If the store evicted the earlier
call's events, it finds nothing and keeps today's behavior.
- Upgrade-safe run state: no persisted shape changes. The check reads
events the Runner already commits. A run saved by the previous release
resumes with the same history and the same answer.
- ACPX and other providers emit no dynamic `tool.execution.*` events, so
the check is a no-op for them.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 10:43:21 -06:00
DottaandClaude Opus 5.5 66a2cbab98 fix(runner): adopt the in-flight Codex turn after a restart lands just after turn.start (#20057)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run survives a server restart by resuming its durable
provider session and adopting the turn that was in flight
> - The adoption rule (RC-59) only adopts a turn named by the latest
`turn.start` command's accepted result
> - Two gaps made it fail. A restart just after `turn.start` leaves that
command `execution_indeterminate` (the runner died after journaling it).
Separately, the rule demanded that Codex echo the requested turn id,
which it never does.
> - The restored session then rejected Codex's own in-flight turn as
unexpected, and the user's Retry failed on the first click
> - This pull request adopts exactly the turn the runner reconciled for
that ambiguous start, and accepts Codex's own turn ids
> - The benefit is that a restart landing right after a turn starts no
longer costs the user a second Retry

## Linked Issues or Issue Description

Found by the restart-chaos campaign: wave `w-20261108T134237-s9401`,
task t77 (Codex C1), Retry run `5358f1cb`. Related: #19852 (fast
follow-up turns), RC-59 adoption. No duplicate PR was found.

**What happened?**

The Retry run submitted its first turn at 14:05:25.171. A K2 SIGTERM
followed at .690 (graceful drain, then `resume_dead_runner`).
- The runner had journaled the `turn.start` and then died, so on restore
it settled the command as `failed / execution_indeterminate`.
- The runner's restored Codex session reconciled the ambiguous start to
the live replacement turn (`session.reconciled`,
`activeProviderTurnId`).
- Codex then sent `turn/started` for that turn. The restored driver had
no active or pending turn, because `controllerAcceptedTurnId()` was
null, and failed with `turn_binding_mismatch`.
- The run ended `native_event_replay_conflict`, and the user needed a
second Retry.

**Expected behavior**

The restored session adopts the runner-reconciled in-flight turn and
finishes it. It does not treat it as a replay conflict.

**Steps to reproduce**

1. Start a native Codex run.
2. Send SIGTERM to the server about 0.5 s after the first `turn.start`,
before runnerd confirms it.
3. Let the server restart and resume the run (`resume_dead_runner`).
4. Before this PR, the run fails with "Provider started an unexpected
turn."

**Paperclip version or commit**

`master` before this PR.

## What Changed

- `controllerAcceptedTurnStartId(commands, options)`
(runnerd-codex-transport.ts):
- **Codex and OpenCode results.** These providers assign their own turn
ids, so an accepted result never echoes the requested `turn_lab_` id,
and RC-59 adoption never matched a real Codex result. For them, the
accepted turn is now adopted only while it is also the runner's live
active turn in its authenticated `session.snapshot`. A turn the restored
runner reports lost has no live turn, so it keeps the established
restart-continuation path. ACPX still requires the echoed id.
- **Indeterminate start.** When the latest `turn.start` (after the
latest attach) failed with `execution_indeterminate`, it adopts the
runner's live active turn. Runnerd reports one only after it reconciles
an ambiguous start to a distinct replacement turn.
- The transport records the snapshot's active turn at `thread/read`,
which `recover()` reads before it evaluates adoption.
- The driver's existing RC-59 guards still apply: the turn must be in
provider history, be the only active turn, and not already be terminal.

## Verification

- Connected restart test, `runnerd-restart-continuation.test.ts` "adopts
the in-flight turn when a restart lands just after turn.start (t77)". It
uses the real runnerd and fake Codex:
  - the runner and provider die after journaling a started turn;
  - runnerd's own restore settles the command `execution_indeterminate`;
  - recovery adopts `provider-turn-1` with no second `turn/start`.
  - It fails without the fix: nothing is adopted.
- Unit cases for `controllerAcceptedTurnStartId`:
  - a Codex-assigned id is adopted only while it is the live turn;
  - an indeterminate start is adopted with the reconciled turn;
- it fails closed with no live turn, an ordinary failed start, a later
attachment, or an ACPX id mismatch.
- Regression guarded: "starts a new turn when the restored runner
reports a lost turn the checkpoint never bound" still passes.
- `runnerd-codex-transport`, `codex-app-server-driver.recovery`,
`runnerd-codex-transport-adoption`,
`runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`
and `runnerd-controller-outage`: 383/383. Runner TypeScript typecheck is
clean.
- Live: the chaos campaign will run the `runner.turn_start.*` crash
points (Codex A1/C1) and a Codex C1 K2 storm, and post the result here.

## Risks

- **Previous-release story.** No persisted shape changes. Commands
journaled by the previous release carry forward and are read with the
new rule. An indeterminate start that the runner did not reconcile to an
active turn still fails closed, as before.
- **Wider Codex adoption.** Codex adoption now actually fires. If
runnerd ever returned an accepted result that names a turn other than
the one this command started, the driver would adopt it. Runnerd binds
the result to the exact command id, so this is the same trust the rule
already placed in ACPX.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 09:26:58 -06:00
DottaandClaude Opus 5.5 946c098bf3 fix(opencode): bind the attached run's contract after a restored launch (#20001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native OpenCode agents run through runnerd and a small OpenCode
proxy. The proxy checks each `paperclip_finish` claim against the run's
completion contract.
> - Each follow-up run has its own contract revision. #19874 lets
runnerd keep the OpenCode process across a contract change, and sends
the new contract on `turn/start`.
> - When runnerd starts again for a follow-up, it restores the saved
session first. The restored proxy launches with the saved run's
contract, but its environment already names the new run.
> - The proxy applied the `turn/start` contract only when the run id
changed. Here the run id did not change, so the proxy kept the saved
contract and refused every correct finish claim.
> - This pull request makes the proxy rebind whenever the turn's
contract differs from the one its session holds. The stale-revision
error also names the expected revision.
> - The benefit: a queued-comment follow-up finishes on its first try,
and an agent that sends a stale revision can repair it.

## Linked Issues or Issue Description

Refs #19874 (warm runs carry their own completion contract).

**What happened**

A user sent a follow-up comment while an OpenCode agent was working.
After the first run finished, Paperclip started a continuation run for
the queued comment. The agent updated the plan and called
`paperclip_finish` with the current revision `2`. The tool refused it
twice with "Semantic result completion contract revision does not
match". The error did not name the expected revision. The agent searched
for it for 3 minutes, and cleanup cancelled the run.

Root cause: runnerd restores a saved `session_open` session before it
applies `run.attach` (`provider_backend.rs`: "execute() restores the
durable provider before dispatching run.attach"). The restore launches
the OpenCode proxy with the saved contract (revision `1`). Runnerd's
environment already names the new run, so the proxy's `PAPERCLIP_RUN_ID`
is the new run. `attach_run_in_place` keeps this provider for a changed
contract (#19874) and sends `paperclipCompletionContract` revision `2`
on `turn/start`. The proxy compared only the run id. The run id matched,
so the proxy skipped `attachRun` and kept revision `1`.

**Expected behavior**

The continuation's first `paperclip_finish` with the current revision is
accepted. If a claim has a stale revision, the error names the expected
revision.

**Steps to reproduce**

1. Use a native OpenCode agent with warm follow-ups (the default).
2. Start a task. While the agent works, post a follow-up comment, so
that it queues.
3. Let the first run finish. Paperclip starts a continuation run with
completion revision `2` and restores the saved OpenCode session.
4. The agent calls `paperclip_finish` with `contractRevision: "2"`.
5. Before this fix, the tool refuses it with "Semantic result completion
contract revision does not match".

The chat eval `steering.discard-queued-message` (runner-opencode) shows
this in run events: `session.resumed` carries revision `2`, and both
`paperclip_finish` results fail with the mismatch.

## What Changed

- `opencode-app-server-proxy.ts`: record the contract that the session
opened with. On `turn/start`, call `attachRun` when the run id changes
or when the supplied `paperclipCompletionContract` differs from the
session's contract.
- `opencode-proxy-task-envelope.ts`: add
`sameOpenCodeProxyCompletionContract`.
- `opencode-server-driver.ts`: the stale-revision error now ends with
`Use contractRevision "<expected>".`, like the criteria error.
- Tests: a real-proxy regression for the restored launch, a driver test
for the new error text, and a `repair-revision` mode in the fake
OpenCode fixture.

## Verification

- `pnpm exec vitest run src/cli/opencode-proxy-input.test.ts
src/drivers/opencode/opencode-server-driver.test.ts
src/cli/opencode-app-server-proxy.test.ts
src/cli/opencode-proxy-thread-read.test.ts` in
`packages/paperclip-runner`: 88 passed.
- Red/green: without the source changes, both new tests fail. `binds the
current contract when a restored launch already names the attached run`
never sees the finish call reach the controller. `names the expected
revision in a repairable stale-revision tool error` does not find the
expected revision in the error.
- `npx tsc --noEmit -p packages/paperclip-runner`: no errors in the
changed files. (My worktree is missing a link for
`@paperclipai/paperclip-eval-kernel`, which causes unrelated errors in
`src/eval/workflow-harness.ts`.)
- End-to-end: the chat eval `steering.discard-queued-message`
(runner-opencode) with this fix cherry-picked. The run link goes in a
comment below.

## Risks

- Previous-release story (`rules/upgrade-safe-run-state.mdc`): this
change does not change a persisted shape. The regression test uses the
saved-state shape: launch parameters from the saved run, environment and
`turn/start` from the new run. A runnerd that does not send
`paperclipCompletionContract` behaves as before, so the proxy keeps its
launch contract. The proxy and runnerd ship in the same runner artifact.
- The proxy can now call `attachRun` for the same run id when the
contract changes. `attachRun` resets the per-run result state. This
occurs only at `turn/start`, when no turn is active, and only when the
contract is different. A result for the earlier contract is not valid
for the new contract.
- The error text changed. No code parses it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 03:35:52 -06:00
Nicky Leach e7e870cfc4 fix(native): decide warm and adopted Runner process reuse from one key (#20005) 2026-11-08 01:02:50 -08:00
DottaandClaude Opus 5.5 65f41593d1 fix(runner): tell agents a skill found in a directory is imported, not re-created (#19983)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can add skills to the company library in two ways: author new
content with `create_skill`, or import an existing skill so its source
is kept
> - #18717 told agents in the `create_skill` contract to import "an
existing skill" instead of copying it
> - An agent read "existing" as "already in the company library"
> - It searched the library, found no match, and saved a managed copy of
a SKILL.md from a vetted local directory
> - The copy has no source locator, so update, audit and reset flows
cannot follow the source
> - This pull request says in the same contract that a SKILL.md in a
local directory, repository, URL or catalog is an existing skill even
when the library has no match
> - The benefit is that agents import such skills and keep their
provenance

## Linked Issues or Issue Description

Refs #18717

**What happened**

A person asked the CEO agent: "Find a useful accessibility-review skill
and add it to our company." The conversation pointed to a vetted local
directory with a SKILL.md. The agent called `search_skills`, found no
match, and wrote in its reasoning "No existing accessibility skill. Now
create it via paperclip_create_skill". It saved a `managed` skill with
`sourceLocator: null`. It did not call the import API that the
`create_skill` description recommends.

**Expected behavior**

The agent imports the skill from its directory through `POST
/api/companies/{companyId}/skills/import`. The saved skill is
`local_path` with the directory as its source locator.

**Steps to reproduce**

1. Put a SKILL.md in a directory that a company project workspace
allows.
2. Give the directory path to the agent in chat, and ask it to add the
skill to the company.
3. Look at the saved skill. Before this change, the agent can save a
`managed` copy without a source locator.

## What Changed

- Add one sentence to the `create_skill` description in
`packages/paperclip-runner/src/protocol-actions/create-skill.ts`: a
SKILL.md in a local directory, repository, URL or catalog is an existing
skill, even when the company library has no match, so import it instead
of copying its content.
- Regenerate the semantic action catalog, the semantic tool contracts,
the protocol manifest digest and the seeded catalog digest.
- Extend the existing contract test so that the discovered, live and
scenario descriptions all contain the new rule.

## Verification

- `pnpm exec vitest run
src/semantic-tools/paperclip-semantic-tools.test.ts -t "distinguishes
authored skills"` in `packages/paperclip-runner` passes. The new
assertion fails on the parent, because the parent description does not
contain the sentence.
- `node scripts/generate-semantic-action-catalog.mjs --check`, `node
scripts/generate-protocol-manifest.mjs --check` and `node
scripts/generate-semantic-contracts.mjs --check` pass.
- End-to-end chat eval `skills.find-and-install` (runner-opencode,
local), with this commit cherry-picked onto the eval branch: before, run
42448994428 failed because the agent saved a managed copy. After, run
42450246200 passed. The agent quoted the new sentence in its reasoning,
imported the directory through the import API, and the saved skill is
`local_path` with the vetted directory as its source locator.

## Risks

- Low risk. The change is model-facing text only. It does not change
permissions, routes or tool inputs.
- Agents that author truly new skills still use `create_skill`. The
sentence applies only when a SKILL.md already exists somewhere.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:47:30 -06:00
DottaandClaude Opus 5.5 051834bdc8 feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - Random restart-chaos waves found these 20 ms–1 s windows only by
luck
> - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable
write boundaries
> - Nothing yet runs a task, kills the server at a chosen point,
restarts it, and checks the plan's invariants
> - This pull request adds that on-demand suite, as a local command and
as a manual CI workflow (not a required check, per Decision 2)
> - The benefit is a repeatable work list: which boundary breaks which
invariant, for Phases 2–4

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only
reconciler used for I1–I7). Stacked on #19938 (call sites), which is
stacked on #19937 (helper).

**Problem**
The named crash points cannot be exercised end to end. No command arms
one point, runs real agent tasks across it, restarts, and grades the
result.

**Expected behavior**
`node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters
<list>` runs one wave per selected point and reports each point as pass,
fail (with failing R rules and violated invariants), not reached, or
harness error. A `workflow_dispatch` workflow runs the same thing on
demand.

## What Changed

- **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the
shared helper and the runner mirror. Each hit reads `<token>
<name>[:<nth>]`, and a new token resets the count. It is test mode only:
`NODE_ENV=production` ignores it, and the disarmed path is still one
comparison.
- **`tests/restart-chaos/main.ts --crash-point
<a>:<nth>,<b>:<nth>,...`** (config, types, ledger):
  - one booted server and no Vite UI;
- per point: take an invariant baseline, launch the point's batch, arm
the point, then on rest disarm it and check invariants as violations
that are new since the baseline;
  - every task is graded R1–R6 and T;
  - `crash-point.json` (v2) holds one entry per point.
- **`tests/restart-chaos/supervisor.ts`:**
  - every spawn gets the control file;
- a SIGKILL whose receipt names the child's pid is restarted like K1:
the control file is emptied first, the owned tree is killed, and a
replacement starts. A failed tree cleanup fails the fault;
- the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a
`.paperclip/.env` above the checkout otherwise suppressed all
scheduling);
  - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`.
- **`tests/restart-chaos/invariants.ts`:**
- the Phase 1 report-only pass with `record: false`: two real-clock
passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7;
  - spool kinds are read from the wave's instance;
  - an unfinished pass is a harness error;
  - each violation carries its wake and run state as evidence.
- **`tests/restart-chaos/crash-points.ts` and
`scripts/crash-points.mjs`:**
- select points by glob, kind and environment, and run them in one wave;
  - a per-point outcome: pass, fail, not_reached or harness_error;
  - `SUMMARY.md` with a violation evidence column.
- **`.github/workflows/crash-points.yml`**, manual only, under the
paid-workflow contract:
  - a credential-free `authorize` job;
  - a `suite` job in `runner-e2e-paid` whose first step reauthorizes;
  - a frozen install with `--ignore-scripts` and no shared cache;
  - only the selected adapters' keys;
  - the workflow is added to both security-test allowlists.
- **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and
"Built (Phase 5)" plus the first work list in the plan.

## Verification

- Focused tests:
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/crash-points.test.ts`;
- `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15
tests, including runtime arming in a real child process;
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72
tests;
  - `tests/runner-e2e/workflow-security.test.ts`: 19 tests.
- Typecheck: `node scripts/typecheck-test-projects.mjs
tests/restart-chaos` passes.
- Loop line: question "how long is one crash-point loop?" | loop: `node
scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min
end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s,
rest about 2 min | result: one point alone takes about 3 min. Before
this change it took about 15 min (one cold boot per point, Vite UI, host
at load 77).
- Full local run, 14 points, codex A1, 11.8 min in all:
- every point was reached (the server killed itself at each one, and the
replacement came back);
  - 11 pass;
- `workspace.finalize.after_start_before_copyback` and
`workspace.finalize.after_copyback_before_done` fail I2 and R6: a
`workspace_finalize` operation stays running after the run succeeded;
- `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its
evidence shows the wake failed with "…retrying once", and the retry
delivered the work, so this is the known refusal-regex false positive.
- The 6 Daytona-only points were not run.

## Risks

- Test harness and an on-demand workflow only. No product code changes.
- `.github/**` needs a CODEOWNER review. The workflow uses provider
keys, so it refuses non-default branches and actors that are not on the
allowlist, like `runner-full-stack-e2e.yml`.
- `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only
crash-point waves. A chaos instance is never a worktree dev instance, so
scheduling must not be suppressed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:41:15 -06:00
DottaandClaude Opus 5.5 c0c26188db fix(runner): stamp event generations and record an earlier generation's events as history (#19960)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through the Paperclip Runner, which turns
provider output (Codex, OpenCode, Claude through ACPX) into durable
runner events
> - In the restart chaos campaign, events from an earlier turn or runner
generation failed the current turn about ten times (RC-15, 20, 20b, 20c,
33, 33b, 59 and 61)
> - #19718, #19722, #19772 and #19783 each special-cased one variant,
and other variants still fail the run
> - The durable-ownership plan (section G, invariant 6) asks for one
rule: every event carries its generation and turn, and an earlier
generation's events are recorded as history and never fail the current
turn
> - This pull request stamps the runner generation on every event and
applies one stale-event rule in the ACPX provider state, the Codex
driver (which Runnerd also uses for OpenCode) and the pending
`turn.start` check
> - The benefit is that a late report about a settled or restored turn
no longer stops a healthy turn, while events for a turn the runner never
saw still fail closed

## Linked Issues or Issue Description

Refs #19718, #19722, #19772, #19783

**What happened?**
A runner or provider can report a turn after that turn settled: a
resumed provider reports the lost turn's usage or terminal, an ACPX
sidecar emits a last delta after the turn terminal, or a held failure of
the previous turn sits in front of a new `turn.start`. Several of these
paths fail the current turn: ACPX `AcpxEventScope` rejects any event for
a non-active turn ("named a stale turn"), the Codex driver fails
`turn_binding_mismatch` for a settled turn's late item between turns,
and `runnerdPendingTurnStartFailure` reports any held non-recoverable
failure, whatever turn it names. Runner events carried no generation.

**Expected behavior**
An event for a turn that is not active, and that an earlier generation
accepted or this generation already settled, is recorded as history and
never fails the current turn. An event for a turn the runner never saw
still fails closed.

**Steps to reproduce**
ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event`
for turn 1. Before this change, the provider state returns an error and
the run fails.

## What Changed

- Runner (`durable/state.rs`, `durable/runner.rs`): every event payload
carries `generation`, the runner process generation that emitted it. A
runner process that recovers state starts the next generation
(`begin_recovered_generation`); it is saved with the next normal write,
so recovery adds no write before the first turn. Loading stays pure. Old
saved state loads as generation 0. Outbox events keep their original
generation.
- PRP event schemas v1–v3 declare the optional `generation` property;
generated validators and the manifest are regenerated.
- ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that
names a settled turn of this run becomes a bounded
`stale_generation_event` diagnostic (32 per state) and changes no state.
Unknown turns still fail.
- Codex driver (`codex-session-state.ts`,
`codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the
turn is not active, and it was restored from the checkpoint (an earlier
generation) or settled in this generation. The classifier's stale-turn
branch, the resume-usage branch, restored-terminal replay and the main
fail point (`notificationNamesActiveTurn`) all use it. A terminal for an
earlier generation's turn is history even when it disagrees with the
checkpoint. A terminal that contradicts a turn this same generation
settled is still a `conflicting_turn_terminal` integrity failure, and a
provider that starts a settled turn again is still rejected.
- Transport (`runnerd-codex-transport.ts`): a pending `turn.start`
ignores a held failure that names another provider turn.
- `DURABLE_TRANSPORT.md` documents the stamp and the rule.

## Verification

- `cargo test -p paperclip-runner-core`: new
`records_late_events_for_a_settled_turn_as_history` (red before),
`stamps_every_event_with_the_runner_process_generation`; two existing
ACPX tests now assert "recorded as history, no state change" instead of
"rejected". Durable tests: 112 passed. `cargo fmt` clean.
- `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests):
late item, delta, terminal and workspace change for a settled turn
during and before the next turn, an earlier generation's conflicting
terminal after recovery, and an unknown turn that still fails. Red on
master: 8 of 9 (the during-turn cases on master already dropped the
event, with a different diagnostic; the between-turn and recovery cases
failed the session).
- `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's
held failure no longer fails the pending start; the pending turn's own
failure and a failed run terminal still do.
- `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787
passed; 2 Codex executable-resolution tests timed out under host load
and pass when run alone.
- `tsc -p tsconfig.json` for the runner package, `check:protocol-types`,
`check:replay-goldens`: clean.
- Not run: chaos waves (the coordinator schedules them on this host).

## Risks

- A late report about a settled turn is now dropped with a diagnostic
instead of failing. A provider that keeps working on a settled turn
could have its output ignored silently; a provider that starts a settled
turn again, or names a turn the runner never saw, still fails closed.
- `generation` is a new optional field. Consumers that do not know it
ignore it, as the PRP contract requires.
- The generation counter is written with the next normal state save, so
there is no new write before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:16:00 -06:00
DottaandClaude Opus 5.5 fda58d8cc1 fix(chat): post the complete final reply, not a 4,000-character run-log preview (#19971)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in chat. The agent's final message becomes the
chat reply.
> - The runner cuts every completed agent message to a 4,000-character
run-log preview. The server then posts that preview as the reply.
> - A long answer, for example a list of 501 open tasks, reaches the
person cut off with "…[truncated]".
> - PR #19959 fixed only the case where the accepted summary starts with
the exact preview text. It missed runs where the final prose differs
from the summary.
> - This pull request keeps the complete final answer in the runner, and
makes the server never prefer a preview over a complete source.
> - The benefit is that the person receives the agent's complete reply.

## Linked Issues or Issue Description

Supersedes #19959.

**What happened**

A chat agent wrote a final answer of about 4,985 characters (all 501
tasks). The posted chat reply was a 4,000-character preview that ended
with "…[truncated]".

**Expected behavior**

The person receives the agent's complete final message. A preview of a
message is never posted when a complete text exists.

**Steps to reproduce**

1. Ask a chat agent (OpenCode or Codex) to list a large set of open
work, so that its final answer is longer than 4,000 characters.
2. Let the run finish.
3. Read the chat reply. It ends with "…[truncated]" at 4,000 characters.

Root cause:

-
`packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs`:
the codex/OpenCode facade projection of `item/completed` and the ACPX
`AssistantMessage` projection cut the final answer with
`bounded_text(text, MAX_TEXT_CHARS)` (4,000 characters).
- `durable/state.rs` `sanitize_value` caps every string again at 4,096
bytes before the event goes to the server.
- `server/src/services/heartbeat-run-summary.ts`
`resolveHeartbeatRunResponse` gives an explicit `final` agent message
top precedence, so it posts the preview.

## What Changed

- Runner (`provider_events.rs`): a completed `final_answer` agent
message keeps its complete redacted text, up to a new 32,000-character
`MAX_FINAL_AGENT_MESSAGE_CHARS` bound. Progress prose and in-flight
items keep the 4,000-character preview.
- Runner (`durable/state.rs`): `preserve_final_agent_message_text` keeps
the complete redacted text of a completed final agent message when it
fits in half of the transport frame. Otherwise the marked preview stays,
so a large message never fails the frame check.
- Server (`heartbeat-run-summary.ts`):
`readCompletedAssistantMessageCandidate` marks a stored preview as
`bounded`. The resolver never posts a bounded preview while a complete
source exists. It posts a marked adapter final response or the accepted
semantic summary instead
(`bounded_final_agent_message_replaced_by_semantic_summary`). It posts
the marked preview only when no complete text exists
(`bounded_final_agent_message_without_complete_source`). This also fixes
replies from runners that are already deployed.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-summary.test.ts`: 86 passed. Six new tests:
a complete final message longer than 4,000 characters is posted in full;
the old exact-prefix case posts the complete accepted answer; a preview
whose prose differs from the summary posts the complete summary (also
with chat precedence); a marked adapter final response wins over a
preview; the marked preview is posted only when nothing complete exists;
a stored preview candidate is marked `bounded`. Five of the six fail
against the `master` resolver.
- `cargo test -p paperclip-runner-core --lib -- final_answer
complete_final`: 3 passed. Both new runner tests fail when the fix is
reverted.
- `cargo test -p paperclip-runner-core --lib`: 581 passed, 3 failed. The
failures are `task_memory::tests::monitor_*` (OOM-kill monitor timing
tests). They do not touch the changed code.
- `cargo test -p paperclip-runner-core --test acpx_event_projection
--test acpx_rich_events --test local_integrity_boundary_golden --test
acpx_provider_events`: all passed. `cargo fmt --check` passes.
- Evidence from the failed eval run: the `item.completed` final
`agentMessage` text was exactly 4,000 characters (3,988 retained plus
the marker). The `run.result.proposed` summary was complete at 4,985
characters.

## Risks

- Larger durable events: a final answer can now be up to 32,000
characters (at most about 128 KB). The outbox keeps the preview when the
text exceeds half of the frame size, so the frame check does not fail.
- Redaction stays the same. The complete text goes through
`redact_sensitive_text_values`, which the preview also used.
- The server treats any final message that ends with "…[truncated]" as a
preview. If an agent writes that text itself, the server posts the
complete accepted summary instead. That summary is also the agent's own
complete answer.
- The runner part reaches users only after a runner release. The server
part fixes replies at once when an accepted summary exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:01:00 -06:00
DottaandClaude Opus 5.5 f755361ffe fix(recovery): deliver a steered message the restart lost before the model read it (#19962)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can steer a running Codex or Pi turn: a comment goes into the
live turn instead of waiting for the next run
> - The provider acknowledges the steer at once, but the model reads it
only at its next step, after the in-flight tool call
> - A control-plane restart in that window kills the provider with the
message in memory. Restart recovery starts a continuation turn that does
not carry the message. The message is lost and no error shows
> - The restart chaos harness showed this in 6 of 6 steered runs (Codex,
K1, K2 and K6, 2 samples each): steer acknowledged, run succeeded, no
reply to the steered message
> - This pull request makes restart recovery deliver every acknowledged
steer that the model did not read, exactly once
> - The benefit is that manual Steer no longer loses messages on
restart, and the planned automatic steering can ship on top of it

## Linked Issues or Issue Description

**What happened?**
A comment steered into a running Codex turn got no reply when a
control-plane restart (SIGKILL, SIGTERM or crash loop) happened during
the agent's tool call. The run succeeded, and the steered comment had no
effect.

**Steps to reproduce**
1. Start a Codex task that runs a 40-second command.
2. During the command, post a comment and click Steer. The provider
acknowledges it.
3. Restart the control plane before the command ends.
4. Recovery resumes the turn with a continuation. The agent finishes the
task and ignores the comment.

**Expected behavior**
The agent reads and handles the steered comment once, after recovery.

**Root cause**
The provider keeps acknowledged steering input in memory until its next
model step. The restart continuation
(`paperclip.native-continuation.v1`) only told the model to continue the
current request, so the acknowledged input was gone.

## What Changed

- `unreadSteeredMessageIds` (runner package): derives
acknowledged-but-unread steers from run-log events in order. A
`steering_acknowledgement` item adds the message id (once per id). A
`userMessage` item in the same turn reads the oldest unread steer. Codex
records each steered input as one `userMessage` item when it enters the
conversation. A `userMessage` item in a later turn reads all earlier
steers, because the continuation carries them. `turn.completed` reads
all steers.
- `readUnreadSteeredMessages` (server): reads those facts from
`heartbeat_run_events` (only transport-validated events with a
`sourceEventId`). It also reads the comment or interaction text for the
same company and issue. The run log is the durable record of both
"acknowledged" and "read", so no new column is needed.
- The restart continuation now includes `steeredMessages` and one
instruction to handle them, oldest first. The runtime reads them through
a lazy `getUnreadSteeredMessages` option, only when a restart
continuation starts. Native recovery wires it for every run.
- `queuedInteractionSteeringBody` is extracted, so recovery rebuilds a
steered interaction response with the same text the steer sent.
- Fake Codex gets `--consume-steer`, which emits the `userMessage` item
for an accepted steer, like real Codex.

## Verification

- Red/green live test with real runnerd and fake Codex
(`runnerd-restart-continuation.test.ts`, "delivers a steered message the
restart lost exactly once"): start a held turn, steer it, SIGKILL runner
and provider, restore and recover.
- "unread": the continuation carries the message exactly once. Without
the runtime change this test fails.
- "read" (`--consume-steer`): the continuation does not carry it again.
- Both: after the continuation completes, a later restart has nothing to
deliver.
- `steering-delivery.test.ts`: ordering, dedupe, initial-input,
later-turn and completed-turn rules.
- `unread-steered-messages.test.ts` (embedded Postgres): ignores events
without a source id and messages from other issues. It reads interaction
text, ignores model-output items as read evidence, and returns nothing
after the continuation input.
- `pnpm vitest run src/steering-delivery.test.ts
src/live/runnerd-restart-continuation.test.ts` (runner) and `pnpm vitest
run src/services/native-runtime/unread-steered-messages.test.ts`
(server): pass on the rebased revision.
- `issue-queued-comments-routes.test.ts` and
`native-session-runtime.test.ts -t restart`: pass.
`native-session-resume.test.ts` "archives a damaged prior epoch…" fails
locally on plain `origin/master` too (local runnerd binary); CI runs it
with a fresh build.
- Restart chaos (Codex steer mode, K1/K2/K6) runs after this lands in
the harness slot. Before: 0 of 6 steered messages answered.

## Risks

- Duplicate delivery: a provider that records steered input without a
`userMessage` item, or a `userMessage` event lost in the crash window
before persistence, makes recovery send the message again. The model
then sees it twice. That is safer than losing it, and the instruction
asks the model to handle each one once.
- Pi (ACPX) steering has no `userMessage` read evidence today, so a Pi
steer that the model already read before a restart may arrive again in
the continuation.
- Only the restart continuation carries the messages. Other recovery
paths start after a completed turn, which already read every steer.
- No schema or migration change. A session saved by an earlier release
resumes normally; its run log already holds the acknowledgement events.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 22:22:52 -06:00
DottaandClaude Opus 5.5 18644a3263 feat(recovery): instrument 20 durable write boundaries with crash points (#19938)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - #19937 added `crashPoint(name)` and a registry of 20 named
boundaries from the Phase 0 inventory
> - A name without a call site cannot fire, so the crash-point suite
cannot test it
> - This pull request puts one call at each registered boundary and adds
a test that keeps the registry and the call sites in step
> - The benefit is that the on-demand suite (next pull request) can kill
the control plane at every known write-ahead gap

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead
coverage"). Stacked on #19937 (helper and registry). Refs #19861
(ownable kind ids).

**Problem**
The registered crash points have no call sites.

**Expected behavior**
Each registered point is called at its gap: after the first durable
write or outside effect returns, before the next one starts. Disarmed
(always in production) each call is one comparison.

**Scope**
Call sites and a coverage test only. The suite runner and CI workflow
follow.

## What Changed

- Server call sites (`@paperclipai/shared/crash-points`):
- `sandbox-creation-operations.ts`: around the provider `create` call
(intent → provider → lease).
- `native-session-executor.ts`: after `native.process_start_requested`,
before spawn.
- `paperclip-control-plane-port.ts`: after the result-accept
transaction, before finalization starts (only when a result was
inserted).
- `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before
the run projection.
- `workspace-operations.ts`: `workspace_finalize` only — after the
running row, and after `settle()` before the done row.
- `environment-runtime.ts`: provider release → lease row; teardown claim
→ destroy; reusable destroy → lease release; stop-and-retain claim →
stop.
- `heartbeat.ts`: lease release → runtime services; budget reservation
hold → dispatch.
- `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`.
- `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit →
spool unlink; receipt spool → DB row → unlink.
- `packages/paperclip-runner/src/crash-point.ts`: a small copy of the
helper for controller code in this package, which does not depend on
`@paperclipai/shared` (adding that dependency would change the runner
content ID). It arms only `runner.*` names. runnerd never sees the
variable: its environment is an allowlist.
- Runner call sites: after the runnerd spawn, before its identity is
published (both spawn paths); `turn.start` result received → controller
journal saved → next send.
- `server/src/__tests__/crash-point-call-sites.test.ts`: every
registered point has a call site, every call names a registered point,
and every `kinds` entry is an ownable kind id of the Phase 1 registry.

## Verification

- `cd server && npx vitest run
src/__tests__/crash-point-call-sites.test.ts` → 3 passed.
- `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter
@paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean.
- Behavior with nothing armed is unchanged: each call returns at its
first comparison (measured in #19937: about 3.7 ns).

## Risks

- Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT`
names it and `NODE_ENV` is not `production`.
- Points on the path before `turn.start` (process start, runner spawn)
cost nanoseconds, within plan invariant 8.
- `usage-receipts.ts`: the persist-then-unlink one-liner became a block
with the same statements in the same order.
- The runner package change gives the runner a new content ID, as any
runner source change does. No persisted run state changes
(rules/upgrade-safe-run-state.mdc).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits) and a read-only exploration subagent that
mapped the boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:56:33 -06:00
DottaandClaude Opus 5.5 0189fedffc fix(native): bound an idle warm local runner whose controller died (#19879)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between
runs so a follow-up turn starts fast. The follow-up default flip (next
PR) makes warm the default for local Codex and OpenCode agents.
> - The controller owns a warm runner's idle window. Runnerd ignores the
idle timeout itself.
> - A graceful controller shutdown suspends idle warm sessions. A
controller that dies (SIGKILL, crash) does not. Local runners launch
with no process reconnect grace, so the idle runner kept running until
its 1-hour connection lease ran out, owned by nobody.
> - This pull request gives warm local runners the 7-minute reconnect
grace that remote runners already use.
> - The benefit: idle runners are reaped by proof within a bounded time,
never left orphaned, before warm becomes the default.

## Linked Issues or Issue Description

Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852.

**What existing behavior does this improve?**
Ownership of an idle warm local runner after its controller dies.

**Current behavior**
`runnerProcessReconnectGraceMs` is `undefined` for local targets, so
runnerd launches without `--reconnect-grace-ms`. An idle warm runner
whose controller was killed stays alive until its connection lease (one
hour) expires. The restart-chaos harness cannot show this: its K1 kills
the controller's whole process tree, runner included.

**Proposed behavior**
`localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7
minutes) for a warm policy and `undefined` for per-turn. After the
grace, runnerd records `transport_reconnect_grace_exceeded`, stops its
provider, keeps durable state and exits. The next run resumes cold.

**Reason and benefit**
The restart plan requires idle runners to be reaped by proof, never
orphaned. Seven minutes is far longer than a controller restart (13–20 s
locally) and the long-outage fault holds (3–5 minutes), so a live turn
still reattaches.

## What Changed

- `native-runner-recovery-contract.ts`:
`localRunnerProcessReconnectGraceMs`.
- `native-session-executor.ts`: the local runnerd transport passes it as
`runnerProcessReconnectGraceMs`. Remote targets keep `null`
(lease-owned).
- `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex.
An idle warm runner, after its turn completed, whose controller
detaches.
- `doc/DEVELOPING.md`: the grace and its bound.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the
idle runner exits with `transport_reconnect_grace_exceeded` and the next
transport resumes the same provider session cold. With no grace (`null`,
today's local launch), the runner is still alive after the controller
left.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-recovery-contract.test.ts`:
warm gets the remote grace; per-turn keeps none; the grace outlasts a
controller restart and a long outage.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Local restart measurement with warm runners (K1/K2, codex, claude,
opencode, n = 3 each): every interrupted run recovered, with recovery
latency equal or lower than per-turn. The chaos waves with this change
are in the default-flip PR.

## Risks

- A warm local runner whose controller stays down for more than 7
minutes exits. A turn in progress then resumes through
`resume_dead_runner`, the same path as a controller crash that kills the
runner. Remote runners already behave this way.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted state changes. A warm runner launched by the previous release
keeps the launch arguments it started with (no grace) until it exits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:11:18 -06:00
Can H. Tartanoglu 410aa6d1b4 test(runner): resolve provider fixture signals through PATH (#14476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its native runner tests verify exact provider shutdown and recovery.
> - Four fixture call sites invoke `/bin/kill` directly.
> - NixOS supplies `kill` through PATH, so those tests fail before
checking process state.
> - This pull request resolves `kill` through PATH, as the neighboring
runner tests already do.
> - The same stop, probe, and resume assertions can run on non-FHS
hosts.

## Linked Issues or Issue Description

**What happened?**

Four Codex provider tests fail with `No such file or directory` on
NixOS. Each failure comes from a test-owned `/bin/kill` invocation. The
executable is installed on PATH.

**Expected behavior**

Tests should probe and signal their disposable provider processes with
the installed `kill` command.

**Steps to reproduce**

On a NixOS host with `kill` on PATH and no `/bin/kill`, run the
`codex_provider` Rust integration target. Three `durable_stop_*` cases
and `structured_question_round_trips_through_the_normalized_backend`
fail. All four failures were reproduced separately on unmodified
`e9debd3eac328e6abe87ab89e9ff40ee2bb56b73`.

Related: these failures also blocked the broader local qualification of
#14474. This fixture fix has no dependency on that runtime change.

## What Changed

- Replace four test-only `/bin/kill` executable paths with `kill`.
- Retain every signal, PID/process-group operand, assertion, and unwind
cleanup.
- Match existing PATH lookup in this file and
`tests/process_supervisor.rs`.

## Verification

Rebased onto master `820c458465` (single commit `ad1fb1043`).

- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --locked -j4 --
--test-threads=2`: 94 passed, 0 failed, 2 ignored (subprocess helper
entries that their parent tests run). The four tests that used
`/bin/kill` pass on a host without `/bin/kill`.
- No production code or documentation changes.
- Hosted CI: `General tests (workspaces-b)` timed out in
`packages/db/src/backup-lib.test.ts` and `e2e shard (1/8)` hit
`ERR_STREAM_UNABLE_TO_PIPE`. Neither area is touched by this test-only
diff. Master pushes currently run the "CI profile skipped" workflow, so
there is no master run of those jobs to compare against.

## Risks

Low risk. Tests require a working `kill` on their test PATH. Fixture
PIDs and arguments stay unchanged.

## Model Used

- Anthropic Claude Sonnet 5.5 (`claude-sonnet-5-5`) through Claude Code
with tool use and code execution, for the rebase and verification.
Original change: OpenAI `gpt-6-astra` through OpenCode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-07 13:02:29 -06:00
DottaandClaude Opus 5.5 dcaada5f59 fix(native): carry each warm run's own completion contract (#19874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd and the provider
process alive between runs, so a follow-up message can start fast.
> - Each run has its own completion contract. The revision changes with
each follow-up, and the agent must name that revision when it finishes.
> - On a warm attachment the transport rebuilt the `run.attach` payload
from its constructor options. Those options belong to the first run.
Every later run kept the first run's contract.
> - The OpenCode proxy checks the finish claim locally against that
stale contract. Every warm OpenCode follow-up was refused ("Semantic
result completion contract revision does not match"), and the agent
looped until the run timed out.
> - This pull request carries each run's own contract into the warm
attachment, and lets runnerd adopt it without a cold provider resume.
> - The benefit: warm follow-ups finish correctly, and they keep the
live provider process, which is the reason to be warm.

## Linked Issues or Issue Description

Refs #19852 (startup sub-spans), #19846 (TTFT baseline).

**What existing behavior does this improve?**
Warm cross-run attachment for native Codex and OpenCode agents.

**Current behavior**
In a local measurement wave with `lifecycleMode: warm`, 4 of 4 OpenCode
follow-ups looped for 8+ minutes. Each finish call was refused with
"Semantic result completion contract revision does not match". The saved
`runAttachTemplate` after the attachment still had revision `1`, while
the follow-up turn named revision `2`. Codex hid the defect because its
finish claim goes through the controller's tool path, which uses the
current input.

**Proposed behavior**
The attachment carries the attaching run's contract. Runnerd updates the
contract in place for Codex and OpenCode and keeps the provider process.
The OpenCode proxy receives the current contract with the run identity
on `turn/start`.

**Reason and benefit**
Warm follow-ups are the largest TTFT lever in the baseline. A
correctness-only fix (cold-resume the provider when the contract
changes) would make every warm follow-up relaunch Codex or OpenCode,
because the revision changes on every follow-up.

## What Changed

- `attachRun` carries `completionContract` through `NativeSession`, the
harness backend, the Codex harness session and the runnerd transport.
`executeNativeSession` passes the current input's contract. The
transport uses it instead of its first-run `resumeCompletionContract`.
- Runnerd (`codex_provider.rs`, `attach_run_in_place`): a changed
contract no longer forces a cold resume for `codex` and `opencode`.
Codex app-server never receives the contract, and the OpenCode proxy now
receives it per turn. Other providers keep the cold-resume boundary. A
changed tool set still forces it.
- Runnerd sends `paperclipCompletionContract` with each OpenCode
`turn/start`, next to the existing `paperclipRunId`.
- The OpenCode proxy passes that contract to `attachRun` when the run
changes. `OpenCodeHarnessSession.attachRun` replaces its envelope's
contract and validates it.

## Verification

- `src/live/runnerd-codex-transport-warm-attachment.test.ts` (new case):
real runnerd and fake Codex. After a warm attachment with a new
contract, the saved `runAttachTemplate` has the new revision and the
Codex pid is unchanged.
- `src/cli/opencode-proxy-input.test.ts`: the real proxy bundle serves a
warm second run with revision `proxy-v2`. The second
`paperclip/runResult` is produced and names `proxy-v2`. Without the fix
the finish call is refused.
- Each test fails without its part of the fix: the proxy test without
the proxy change; the transport test without the transport change (stale
saved revision); and, with `PAPERCLIP_ATTACH_TRANSITION_RUNNER` pointed
at a runnerd built without the `codex_provider.rs` change, the Codex pid
changes.
- `cargo test -p paperclip-runner-core --lib warm` (11 tests),
`native-session-runtime.test.ts` (232), the OpenCode driver, backend and
warm-attachment suites, and `tsc --noEmit` for the runner package pass.

## Risks

- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Saved `runAttachTemplate` values carry forward,
and the next attachment writes the current contract. A new runnerd with
an old OpenCode proxy behaves as today: the proxy ignores the new
parameter and keeps the stale contract. An old runnerd sends no
parameter, so a new proxy keeps its launch contract.
- In-place adoption widens the warm-retention path for Codex and
OpenCode to changed contracts. The idle, accounting and pending-work
checks still apply first.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:40:51 -06:00
DottaandClaude Opus 5.5 a9ac2e2039 feat(native): split local runner startup into stage, launch and session-load spans (#19852)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a person messages an agent, the agent must start its turn fast.
Time to first token (TTFT) and agent boot are first-class metrics.
> - The Phase 0 baseline (#19846) found that local runner startup costs
1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT.
> - The run log records that startup as one span
(`runner.session.bootstrap` or `runner.session.resume`). We cannot see
if the time goes to staging, to the `paperclip-runnerd` process launch,
or to the provider session load.
> - This pull request splits local runner startup into three sub-spans
in the run log.
> - The benefit: the follow-up work to cut follow-up-turn TTFT can
measure each step before and after.

## Linked Issues or Issue Description

Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside
local runner startup"). Refs #19838.

**What existing behavior does this improve?**
The run-log performance spans for local native runner startup.

**Current behavior**
The local path records one `runner.session.bootstrap` or
`runner.session.resume` span that covers runtime staging, the runner
process launch and the provider handshake together. The remote path
already records `runner.runtime.stage` and `runner.process.launch`; the
local path does not.

**Proposed behavior**
The local path records three sequential child spans of
`runner.session.startup`: `runner.runtime.stage`,
`runner.process.launch` and `provider.session.load`.

**Reason and benefit**
The TTFT work needs to know which step dominates before it changes the
runner lifecycle. Without these spans, a change to warm runner reuse
cannot show which cost it removes.

## What Changed

- `RunnerdCodexTransport` has a new optional `onStartupStep` callback.
`#start` and `#resume` time three milestones and call it synchronously.
A failing callback never changes startup. A nested startup (exact
adoption from `thread/start`) keeps the outer clock.
- `runner.runtime.stage`: launch preparation before the runner process
exists.
- `runner.process.launch`: spawn until runnerd accepts the run
(`run.prepare`, or the recovery attachment on resume). An adopted live
runner records no launch. This keeps the existing meaning in
`summarizeStartupReuse`: a launch span means a fresh runner.
- `provider.session.load`: `session.open` (or the recovery probe) until
the provider identity is confirmed.
- `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each
step as a `run.performance.span` row with `mode` and `target: "local"`.
It uses `trace.defer`, so the run-log write never delays `turn.start`.
- The native executor passes the callback only for a local target.
Remote launchers keep their own spans, so there are no duplicates.
- `redaction.ts`: allowlist `runner.process.launch` and
`provider.session.load` as native span names.
- `doc/run-log-events.md`: new section "Local runner startup sub-steps".

Data path (AGENTS.md §5.7): run log only, plus Observability spans when
an operator sets an OTLP endpoint. The attributes (`mode`, `target`,
`outcome`) are already in the closed span-attribute allowlist. No
Telemetry change.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd
and the fake Codex fixture. Bootstrap and a cold resume each report
stage, launch and session load in order, with `ok` outcomes and no
overlap. A throwing sink does not change startup.
- `cd server && npx vitest run
src/services/native-runtime/native-run-trace.test.ts
src/__tests__/redaction.test.ts`: the rows have the correct parent,
offsets, durations and failed level. A stalled run-log sink does not
block the caller. The rows settle before
`task.provider_session.measured`. The names survive redaction.
- The existing `runnerd-codex-transport*` suites pass (412 tests). Under
host load (load average 16–27), the first full run had 7 timing failures
in suites that do not set the callback. Each failing file passed when I
ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for
the failing case alone.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- Low risk. Without `onStartupStep`, the transport behaves as before:
the clock is not created and each milestone call is an optional-chain
no-op.
- More run-log rows: three per local runner startup.
- The step boundaries are controller-side observations.
`runner.process.launch` includes the runnerd authentication and the
`run.prepare` acknowledgement, not only the OS spawn.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:57:31 -06:00
DottaandClaude Opus 5.5 b5b972ceb1 fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The local runner (`paperclip-runnerd`) supervises each provider
process, such as the ACPX sidecar for Claude, in its own process group
> - Run teardown stops that process group, and nothing else
> - Claude Code starts each Bash tool command in a new session
(`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job
in that session, and the wrapper exits at once. The job is then a child
of init and outside the owned group
> - In a restart chaos storm, a run failed after a restart resume. Its
tool job kept running beside the task's next run, in the same workspace,
for more than four minutes
> - This pull request makes the supervisor record every process that its
provider starts, including processes that leave the group, and stop them
all before teardown reports success
> - The benefit is that a run's tools cannot outlive the run and cause
duplicate side effects in the next run

## Linked Issues or Issue Description

**What happened?**
A restart chaos storm ran a local ACPX Claude task. After a restart
resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo
"started pid $!"` and then ended its turn without a result. The run
failed with `native_finalization_missing`. The runner tore down the
sidecar group, and the guardian killed the provider group. `bench.sh`
was in the session that Claude Code made for the tool (pgid and sid =
the exited wrapper, ppid = 1). It was never signalled. It kept forking
`sleep 1` while the task's next run started and finished in the same
workspace.

**Steps to reproduce**
1. Run a local ACPX Claude task whose agent runs a Bash tool command
such as `nohup ./bench.sh x > log 2>&1 & echo $!`.
2. End the run (any outcome, including failure after a restart resume).
3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still
running with ppid 1.

**Agent adapter(s) involved**
Claude Code (local ACPX). The gap is in the shared supervisor, so any
provider whose tools leave the group is affected.

**Root cause**
`SupervisedProcess`
(`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`)
spawns with `process_group(0)` and stops only that group in
`terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its
own comment said that a caller that must exclude escaped or re-parented
descendants needs a separate lifetime fence. No caller had one. The ACPX
sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`)
and the ACPX provider guardian (`installation-integrity.ts`, which kills
only its own group) therefore left every `setsid` tool job behind. The
failure path after the resume was not special: the same gap applies to
every teardown.

**Expected behavior**
When a run ends for any reason, every process that its provider or tools
started stops, and teardown proves that they are gone. Identity comes
from PID, group, session and start time, never from a command name.

## What Changed

- Add the `paperclip-process-table` crate. It reads PID, parent, process
group and start time for every process (libproc on macOS, `/proc` on
Linux). `runner-core` forbids unsafe code, so the two checked libproc
calls live in this small crate.
- Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50
ms) records each descendant of the supervised process by PID and start
time. It also records each group and session that a descendant leads,
with the leader's start time. A process belongs to the lineage when its
parent belongs, or when it is in a recorded group or session and started
no earlier than its leader. Groups and sessions that the supervised
process only inherited (runnerd's session) are never recorded. A
recorded ID whose PID now names a different process is dropped.
- `SupervisedProcess` samples the lineage just before it signals, and
after the leader is reaped it stops every live member: TERM, then KILL
after the shutdown grace. It fails with an error until it proves that no
member is alive (grace plus 5 s). This applies to `terminate_group`,
`terminate_group_immediately`, `wait` and `Drop`.
- If the lineage sampler cannot start, the supervisor stops the process
it just started and reports the failure, so a provider never runs
without tool cleanup.
- Table reads and state updates happen under one lock, so an older read
cannot erase members that a newer read recorded.
- The process table reports processes that exist but cannot be read. A
recorded member that cannot be read stays unresolved, and teardown fails
until its exit is proven.
- The Runner workspace now has two members.
`scripts/runner-release-crate-cache.mjs` and
`scripts/runner-debug-crate-cache.mjs` cache and mark current the
process-table lib unit with the Runner crate (otherwise Cargo rebuilds
the Runner crate after every restore). The debug contract accepts only
this library-only member and still refuses any other workspace change.
The workflow cache path lists and the debug clean step are updated to
match.
- `terminate_group_immediately` now accepts `EPERM` from the group
signal when the leader has already exited. macOS returns `EPERM` for a
group whose only member is the unreaped leader. The lineage pass then
proves that no other member remains.

## Verification

- `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job
immediate_group_fence` (macOS, local).
- New real-process tests in `process_supervisor.rs`: a supervised shell
starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs
`nohup <loop> &`, writes the job PID and exits. The test asserts that
the job is outside the owned group and re-parented, then calls
`terminate_group`, `terminate_group_immediately` or drops the process,
and asserts that the job is gone when the call returns.
- Red: with the tracker disabled, all three fail with `tool job <pid>
outlived its provider`.
  - Green: all three pass.
- Unit tests in `process_lineage.rs` cover a re-parented job, a job that
left its group but stayed in the recorded session, a session made after
the first sighting, PID reuse, and no claim through the supervisor's
inherited session.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the
process-table library is cached with the crate, and any other member
still needs review).
- `cargo test -p paperclip-process-table` (includes an exited process
reported as gone), and `cargo test -p paperclip-runner-core --lib` (570
passed).
`acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails sometimes under full-suite load. It fails at the same rate with
the tracker disabled, so it is a flake that already existed.
- Sampler cost: one table read takes about 0.4 ms for 1,200 processes on
macOS (about 0.8% of one core for each supervised process).
- The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did
not run. The local session was not permitted to load provider
credentials. It is not proven against a live Claude run.

## Risks

- Teardown now stops all the processes that a provider started, also
intentional background jobs. This is the intended contract: nothing
outlives the run.
- A wrapper that lives less than one sample (50 ms) and orphans a job
before the sampler sees it is not recorded. Claude Code's wrapper
sources a shell snapshot and lives for seconds. On Linux, a
child-subreaper would close this gap. That is not in this change.
- When `runnerd` itself is killed with SIGKILL, this code does not run.
The provider guardian still kills its own group, but escaped tool
sessions are not stopped in that case.
- Teardown can return a new error when it cannot prove the lineage
stopped. Callers already handle `terminate_group` errors as provider
shutdown failures.
- The server does not yet gate the next run on the tool lineage of the
prior run directly. It relies on the runner finishing its teardown
before the runner exits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:24:41 -06:00
DottaandClaude Opus 5.5 b2af45f0f1 fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.

## Linked Issues or Issue Description

Refs #19516

**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.

**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.

**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.

**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.

I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.

## What Changed

- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
  - The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
  - A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
  - Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.

## Verification

**Red on `master` (e482f0f632).** Behavioural red with the first version
of the new test:

```
 FAIL  src/services/native-runtime/generate-image-tool.test.ts > generate_image for OpenAI API-key Codex agents (N-12) > lets an API-key Codex agent make an image with its existing OpenAI key, saved in its workspace and billed
AssertionError: expected [ 'connections_search', …(51) ] to include 'generate_image'
 ❯ src/services/native-runtime/generate-image-tool.test.ts:78:60
      Tests  1 failed | 3 passed (4)
```

The final test file on master's source also fails. The reason is that
the module does not exist: `Cannot find module
'./generate-image-tool.js'`.

**Real pinned Codex binary.** The new test is
`packages/paperclip-runner/src/drivers/codex/codex-api-key-image-tools.test.ts`.
It uses the runner's isolated launch, the API-key `auth.json` and a
local Responses endpoint. It shows that the model gets no `image_gen`,
`imagegen` or `image_generation` tool, and that a server-advertised
`generate_image` tool does reach the model:

```
 Test Files  1 passed (1)
      Tests  1 passed (1)
```

**Green after the fix:**
`server/src/services/native-runtime/generate-image-tool.test.ts` uses
embedded PostgreSQL and a stubbed OpenAI fetch.

```
 Test Files  1 passed (1)
      Tests  11 passed (11)
```

The 11 tests cover these behaviours:

- **Main path.** The tool is advertised. The request goes to
`api.openai.com/v1/images/generations` with the existing key. The PNG is
written to the workspace. The key does not appear in the result or in
activity. One `cost_events` row is written: 50 text tokens and 1000
output tokens, which is $0.03025, estimated, from the rate card. A retry
and `recover()` replay the receipt with no second fetch and no second
cost row.
- **No binding:** the tool is not offered and `execute` refuses.
- **Budget hard-stop.** The stop pauses the agent, so the call is
refused before any fetch and no file is written.
- **Bad grant:** the call is refused before any fetch.
- **Project budget stop.** The task's project is paused by its budget,
so the call is refused before any fetch.
- **Charge settlement.** A crash left a `charged` receipt. The retry and
`recover()` record the cost once, report the image path and do not buy
another image.
- **Oversized reply.** The read stops before the body is parsed.
- **Save failure.** The spend is still recorded. A same-call retry does
not pay again and reports the failure.
- **Binding gate.** Only Codex on a direct OpenAI API key gets the
binding. A subscription, ACPX, OpenRouter or a routed connection does
not.
- **Rate card math.**
- **Provider errors.** The error text does not contain the key, and
non-PNG data is rejected.

**Neighbouring suites:**

- `src/drivers/codex` in the runner package: 24 files, 350 tests passed.
- Server: `paperclip-runner-tool-authority.test.ts`,
`native-session-resume.test.ts`, `fast-responses.test.ts`,
`ai-connection-session.test.ts` and the new suite. 157 of 158 passed.
- The one failure is `native-session-resume.test.ts › archives a damaged
prior epoch … with real runnerd` (ENOENT on `authority-epochs`). It
fails the same way on unmodified `master` in this local environment, so
this change did not cause it.
- `tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` exits 2 (broad). Required CI supplies the broad
coverage.

## Risks

- **New security surface: this needs maintainer sign-off.** The server
now sends the agent's OpenAI API key to `api.openai.com` on the agent's
request. The tool is limited in these ways:
  - The key stays in the server.
- Each call checks run, task, budget, mutation and connection authority
again.
- The destination name comes from the server and must match
`^[a-z0-9-]+-[0-9a-f]{8}\.png$`. The write is create-only and confined
to the workspace root. On macOS it uses `O_NOFOLLOW_ANY`. On Linux it
uses a `/proc` descriptor walk.
- **Spend.** Each call costs money. The tool description tells the agent
to make only what the task needs. The cost is an estimate from a rate
card (`costStatus: "estimated"`), not an invoice. The rate card applies
the image-output rate to all output tokens, so the estimate errs on the
high side.
- **Checkpoint rotation.** The tool-contract bump from v18 to v19 makes
existing native provider threads start fresh once at their next turn.
This is what the contract requires when the tool surface changes.
- **Pricing drift.** If OpenAI changes `gpt-image-2` prices, the rate
card must be updated.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, sub-agents) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 07:48:39 -06:00
DottaandClaude Opus 5.5 5eac099fbb fix(agents): batch independent follow-up questions into one card (#19634)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents ask users for input with the `request_human_input` tool or
the `ask_user_questions` interaction. Paperclip shows each call as one
question card.
> - In an onboarding walk, a Codex execution agent asked five
independent questions (name, menu and prices, contact, style, output).
It posted five cards, one after the other. Each card came 20-60 s after
the previous answer.
> - The tool description said: "For one question at a time, ask only the
next unanswered question and wait for its real answer." Agents read this
as a rule. The paperclip skill showed only a one-question example and
did not tell agents to batch.
> - No server cap exists. A question set accepts up to 64 questions.
> - This pull request changes the guidance. Agents now ask independent
inputs together in one card. They use a follow-up card only when a
question depends on an earlier answer. One-at-a-time stays available
when the user asks for it.
> - The benefit is fewer waits for the user: one card and one agent turn
instead of five.

## Linked Issues or Issue Description

**What happened?**
An execution agent asked a user five independent follow-up questions in
five separate question cards. Each card appeared only after the user
answered the previous card and the agent finished another turn (20-60 s
each).

**Expected behavior**
When an agent needs several independent inputs, it asks them together in
one multi-question card. The onboarding interview already does this ("1
of 4" pager).

**Steps to reproduce**
1. Complete onboarding on the Codex API path and hand off a task such as
"Create printable cake menu and logo".
2. Watch the execution agent's first turns.
3. See five sequential single-question cards.

**Paperclip version or commit**
master at 4938a930d6 (local dev server).

**Agent adapter(s) involved**
Codex (gpt-5.6-sol). The guidance is shared by all adapters that use the
Runner `request_human_input` tool or the `paperclip` skill.

## What Changed

-
`packages/paperclip-runner/src/protocol-actions/request-human-input.ts`:
replace the ambiguous "For one question at a time…" sentence. The new
text tells agents to ask independent inputs together in one questions
card. They use a follow-up card only when it depends on an earlier
answer. It keeps one-at-a-time for when the user asks for it.
- `skills/paperclip/SKILL.md` and
`skills/paperclip/references/api-reference.md`: add the same batching
rule to the human-input sections.
- `packages/mcp-server/src/tools.ts`: add the batching hint to the
`paperclipAskUserQuestions` tool description.
- Regenerated runner contracts (`semantic-tool-contracts.json`,
`protocol/manifest.json`, `native-execution-seeded.json`) and updated
the legacy MCP alias semantics in `spec/capability/mcp-tool-map.yaml`.
- New tests: `core-contracts.test.ts` (tool description) and
`paperclip-skill-utils.test.ts` (skill and API reference).

## Verification

Red on master (tests added, source unchanged):

```
packages/paperclip-runner: npx vitest run src/protocol-actions/core-contracts.test.ts -t batch
 × tells agents to batch independent questions into one card instead of asking them one by one
AssertionError: expected 'Create a durable human question or ap…' not to contain 'For one question at a time'
 Tests  1 failed | 20 skipped (21)

npx vitest run server/src/__tests__/paperclip-skill-utils.test.ts -t batch
 × tells agents to batch independent human-input questions into one card
AssertionError: expected ' For an open answer, use a text field…' to contain 'ask them together in one card'
 Tests  1 failed | 21 skipped (22)
```

Green after the fix:

```
packages/paperclip-runner: npx vitest run src/protocol-actions src/drivers/runner-tool-bridge.claude-schema.test.ts
 Test Files  4 passed (4)   Tests  46 passed (46)
npx vitest run server/src/__tests__/paperclip-skill-utils.test.ts            Tests 22 passed (22)
npx vitest run server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts   Tests 41 passed (41)
node scripts/generate-semantic-contracts.mjs --check / generate-capability-contract --check / generate-semantic-action-catalog --check / generate-replay-goldens --check: current
```

`packages/mcp-server` has one failure ("allows create issue requests to
omit status…"). It also fails on master without this change, so it is
not related.

This is a guidance change. The automated check proves that the guidance
the agent receives tells it to batch. I did not run a live-model eval.

## Risks

- Low risk. The change is text only: tool descriptions and skill docs.
No schema, API, or storage change.
- An agent could put a dependent question into the same card. The text
tells agents to use a follow-up card for dependent questions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 07:48:28 -06:00
DottaandClaude Opus 5.5 2468b3f907 fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart loses a turn, the native runtime sends
a same-conversation continuation (#19501, #19699, #19771)
> - A restart can also kill shell or background processes that earlier
turns started
> - In three restart storms, the continuation agent could not inspect
its killed `bench.sh` and set the task to blocked
> - The continuation prompt told the agent not to "blindly rerun an
action whose outcome is unknown", so the agent did not re-run the
command
> - This pull request tells the agent that earlier processes may be
gone, and to check the workspace outputs and re-run a command whose
expected output is missing
> - The benefit is that a task can finish after a restart instead of
blocking on a process that no longer exists

## Linked Issues or Issue Description

Refs #19501, #19699, #19771

**What happened?**

Restart chaos workload B5 (Codex and Claude) failed this way three
times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started
`bench.sh` in the background. A restart killed it. The restart
continuation agent tried to find its tool session and got "process
inspection is denied". It then blocked the task. The comment read:
"Blocked by the Paperclip runtime restart: the benchmark session is
unavailable…". It added that it "cannot verify whether the original run
is still active or recover its output, so I have not rerun it".

**Expected behavior**

The agent knows it is in a resumed session after a restart. It checks
the workspace for the command's outputs. If the expected output is
missing, it re-runs the command and finishes the task.

**Root cause**

`packages/paperclip-runner/src/native-session-runtime.ts`,
`restartContinuation`: the prompt did not say that processes from
earlier turns may be gone. It told the agent to inspect each unfinished
command's "current state" and not to "blindly rerun an action whose
outcome is unknown". With the process gone, the agent could not inspect
it, so it blocked.

## What Changed

- The restart continuation prompt now says:
  - This is a resumed session after a Paperclip control-plane restart.
- Shell commands and background processes from earlier turns may no
longer be running, and their tool sessions may be gone. Do not block
only because one cannot be found or inspected.
- Check the workspace for the outputs or artifacts the command should
have produced. If the expected output is present, use it. If the
expected output is missing, re-run the command.
- Do not repeat completed work. Before repeating an action with external
effects (a push, a message, a deployment), check whether it already took
effect. If you cannot tell, report the blocker.
- The prompt does not say that re-running is always safe. Process
teardown (#19727) has not merged.
- `doc/execution-semantics.md`: one additive sentence.

## Verification

- `native-session-runtime.test.ts`: the three restart continuation tests
(Codex, ACPX, OpenCode) now check the new guidance. They were red before
the prompt change and are green after it.
- Guard: "uses the admitted session cwd in the task envelope" now checks
that fresh turns and ordinary resumed turns do not contain the restart
guidance.
- `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/
src/live/runnerd-restart-continuation.test.ts` in
`packages/paperclip-runner`: 603 passed.
- `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for
`packages/paperclip-runner` are clean.
- Not run: a live restart-chaos replay with a real model.

## Risks

- Low. This changes only the prompt text of the restart continuation.
Ordinary turns do not change.
- An agent may now re-run a command whose earlier run had partial side
effects. The prompt asks it to check external effects first, and to
re-run only when the expected output is missing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 03:46:20 -06:00