mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
1b52f359dd35e09541184dfd3df1e896ff70ead4
567
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ff4ba7d319 |
perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description **What existing behavior does this improve?** Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. **Subsystem affected** Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). **Current behavior** Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. **Proposed behavior** When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. **Reason and benefit** See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - **Root checks at startup.** The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - **Per-artifact checks, on the open descriptor.** After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - **Admitted artifacts** become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - **This change needs review from a security owner before merge.** It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - **Misconfiguration fails closed.** A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - **Residual trust.** On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - **Allowlist.** Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
60a60a0cd1 |
fix(opencode): attribute unreadable tool-result images to the tool (#20399)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner drives OpenCode agents. It sends their
OpenRouter and Paperclip-connection model calls through a local provider
proxy.
> - An agent often takes a screenshot of its own work and reads it to
verify the result.
> - OpenCode 1.18.34 cannot put images in OpenAI-compatible tool
results. It moves them into a synthetic user message ("Attached media
from tool result:"). For a text-only model, it replaces the image with
"ERROR: Cannot read image (this model does not support image input).
Inform the user."
> - The model reads that text as a message from the person. In a chat
eval, a DeepSeek worker redid its verification for about 5 minutes,
registered its deliverable twice, and ran out of the 15-minute attempt
clock.
> - This pull request moves that notice into the tool result that
produced the image, worded as a runner note, and removes the synthetic
user message.
> - The benefit is that a text-only agent sees a clear tool outcome and
continues its work. It does not mistake the notice for the person.
## Linked Issues or Issue Description
**What happened?**
An OpenCode agent on `openrouter/deepseek/deepseek-v4.1-flash-1017` read
a PNG screenshot that it made. The next model request contained a
user-role message with the text "Attached media from tool result: ERROR:
Cannot read image (this model does not support image input). Inform the
user." The model reasoned "The previous input from the user was ..." and
repeated finished work.
**Steps to reproduce**
1. Run OpenCode 1.18.34 with an `@ai-sdk/openai-compatible` provider and
a model that has no image input.
2. Have the model call the `read` tool on a PNG file.
3. Inspect the next chat-completions request. The last message is the
synthetic user message above.
**Expected behavior**
The model must get the notice as part of the tool result, not as a
message from the person.
**Environment**
Paperclip runner, OpenCode 1.18.34 (qualified pin), OpenRouter and
Paperclip-connection model routes.
## What Changed
- Add
`packages/paperclip-runner/src/drivers/opencode/tool-media-attribution.ts`.
It finds the OpenCode synthetic attachment message only when it directly
follows tool results. It moves each "Cannot read ..." or "empty or
corrupted" notice into the last tool result of that step as a runner
note. It keeps readable media in the synthetic message. It removes the
message when no media remains.
- The OpenCode provider proxy applies this rewrite before it forwards a
request. It re-serializes the body only when something changed.
- Messages written by a person are never changed. A person who attaches
an image to a text-only model still gets "Inform the user", which is
correct for that case.
- Add unit tests for the rewrite and an integration test through the
real provider proxy.
## Verification
- Red: the new proxy test failed before the change (`expected ... to
have a length of 4 but got 5`).
- Green: `npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts
src/drivers/opencode/tool-media-attribution.test.ts` (82 tests pass).
`tsc --noEmit -p packages/paperclip-runner` is clean.
- Real binary: I ran the real OpenCode 1.18.34 binary with a mock
OpenAI-compatible server and had the model read a PNG. The captured
request has exactly the message shape that the test fixture uses.
- `pnpm verify` returned exit 2 (test files outside the resolved
graphs). Required CI supplies broad coverage.
- A paid live run of
`chat-features.runner-opencode.local.task-coordination.delegate-with-context`
will follow in a PR comment.
## Risks
- Low risk. The rewrite matches the exact OpenCode 1.18.34 strings. If a
future OpenCode version changes them, the rewrite does nothing and the
old behavior returns. It does not fail the request.
- The rewrite is deterministic, so prompt-cache prefixes stay stable
across requests.
- No persisted state changes. OpenCode history stays as it is. Sessions
saved before this change also get the rewrite when they resume.
- Providers that do not use the runner provider proxy are not covered by
this change.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b2a7d322ed |
test: preserve runner commit stall acknowledgement invariants (#20383)
Assert that stalled event commits do not advance acknowledgement or committed state. Clarify the limits of historical failure attribution. Preserve the production implementation already merged in #20385. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
5858cef8f9 |
perf(runner): poll the provider right after an ACK releases it (#20374)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner (`paperclip-runnerd`) sends provider events to the controller over one authenticated socket > - Its main loop reads one control frame, then polls the provider only after a read finds no frame > - After the controller ACKs the last sent event, the next read waits the full 250 ms `RUNTIME_READ_TIMEOUT` before the Runner polls the provider again > - A Codex turn start needs one or two such rounds before the Runner reads `turn/started`, so every Codex turn waits 250 ms or 500 ms for no reason > - This pull request reads only the frames that are already queued after a consumed control frame, and then polls at once, when this connection produced the unacknowledged output > - The benefit is 250 ms less time to `turn/started` for each ACK round, and faster event delivery after each ACK ## Linked Issues or Issue Description This is an enhancement. The fields follow `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** Codex turns wait for the Runner read timeout before the Runner sends `turn/started`. **Subsystem affected** The Runner (`paperclip-runnerd`) durable loop in `packages/paperclip-runner/runner/crates/runner-core/src/durable/`, for the local Codex provider. **Current behavior** On a hosted test deployment (48 h, 114 Codex turns), `runner.transport.started_event` for Codex has two modes: | Codex turn | n | `started_event` p50 | | --- | --- | --- | | no provider cold start in the trace | 57 | 468 ms (all < 700 ms) | | provider cold start in the trace | 57 | 918 ms (51 of 57 ≥ 700 ms) | For Claude (ACPX) turns, the p50 is 90 ms. ACPX returns `turn.started` with the command result. Codex reads `turn/started` from the provider in a later poll. **Proposed behavior** The Runner polls the provider as soon as the controller ACK releases the poll gate. The time to `turn/started` then follows the controller round trip, not the read timeout. **Reason and benefit** - Real Codex (`codex app-server` 0.157, local) sends `turn/started` 2–16 ms after its `turn/start` response. The provider is not the cause. - `runner.transport.started_event` starts when the `turn.start` command result arrives. It ends when the controller commits `turn.started`. In this time the Runner sends `turn.accepted`, waits for its ACK, waits 250 ms, polls, and sends `turn.started`. - On the first turn after a provider launch, the first poll returns only the launch diagnostic that the executor retained. A second ACK round and a second 250 ms wait come before the poll that reads `turn/started`. This is the difference between the two modes above (about 450 ms: 250 ms wait and one event round). - So 250 ms (warm turn) or 500 ms (first turn) of each Codex `started_event` is the read timeout. The rest is one or two controller event rounds (about 200 ms each on that deployment), less than 20 ms of Codex time, and the 10 ms pump interval of the TypeScript transport. - The expected benefit is 250 ms less per warm Codex turn start and 500 ms less for the first turn after a provider launch, plus faster Codex event delivery after each ACK. **Breaking changes** None. No protocol, schema, or controller change. ## What Changed - `AuthenticatedTransport::receive_queued_json` reads with a 1 ms timeout and then restores `RUNTIME_READ_TIMEOUT`. - New `CommandExecutor::provider_poll_is_live` hook (default `false`). The local Codex executor returns `true` only when it owns a running provider with no open startup attempt, so a poll cannot restore or launch a provider. The native executor forwards it for the local Codex facade only. - In the steady-state loop, after a consumed control frame, the Runner uses `receive_queued_json` when it has sent output that this connection produced (a command or a provider poll) and the executor reports a live provider. If no frame is queued, it polls the provider at once. - When a provider poll returns no output, or when no frame was consumed, the Runner uses the 250 ms read as before. A 1 ms read follows only a consumed frame, so an idle loop does not spin. - The replay boundary is the highest source sequence before the welcome's pending commands run. A fresh command delivered in the welcome counts as this connection's output. Replayed commands only resend their stored results. - Replayed output from an earlier connection or process, ACPX, and the other providers keep the 250 ms read. The controller can still send its next command before a poll restores a provider. (Without the live-provider gate, six ACPX source-recovery tests in `runnerd-acpx-source-reconciliation.test.ts` failed: the fast poll restored the ACPX sidecar before the next recovery command.) - Control frames that are already queued behind the ACK are still read and executed before the provider poll. The ACK gate in `poll_executor_events_after_controller_ack` is unchanged. ## Verification - `cargo test -p paperclip-runner-core --lib durable::` (121 passed). Five new tests in `src/durable/tests/ack_release_poll.rs`, included by the transport test module like `hosted_retention.rs`: - `acknowledged_output_polls_the_provider_without_waiting_out_the_read_timeout`: the provider poll follows the ACK read in less than `RUNTIME_READ_TIMEOUT`. Without the change it fails (the poll waited 280–356 ms). - `control_queued_behind_an_ack_runs_before_the_next_provider_poll`: a command sent in the same write as the ACK executes before the provider poll. - `welcome_command_output_polls_the_provider_without_waiting_out_the_read_timeout`: the output of a fresh command in the welcome also releases the poll at once. With the replay boundary taken after the welcome commands, it fails. - `replayed_output_ack_keeps_the_read_timeout_before_the_provider_poll`: after a reconnect, a command sent 50 ms after the ACK of replayed output executes before the provider poll. Without the replay condition it fails. - `restorable_provider_ack_keeps_the_read_timeout_before_the_provider_poll`: when the executor does not report a live provider, a command sent 50 ms after the ACK executes before the provider poll. - `npx vitest run src/live/runnerd-` in `packages/paperclip-runner` with the new debug `paperclip-runnerd`: run with `--no-file-parallelism`. Before the welcome-boundary fix: 576 passed, 3 skipped. After it: 575 passed, 3 skipped, and 1 failure once (`stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`, a SIGKILL timing test) while `cargo test` ran at the same time. `runnerd-codex-transport.test.ts` then passed 293 of 293 three times with this build, and three times with the base build. - Local before and after measurement with the real Runner, the TypeScript transport, and the fake Codex app-server (8 turns, 2 runs each, debug build): | span | before p50 | after p50 | | --- | --- | --- | | `runner.transport.started_event` | 377 ms (first turn 682–695 ms) | 119–122 ms (first turn 117–119 ms) | | `runner.transport.turn_start` | 462–471 ms (first turn 775–780 ms) | 204–212 ms (first turn 247–251 ms) | - The saving on the hosted deployment is not measured. From the analysis above, the expected saving is about 250 ms for a warm Codex turn and about 500 ms for the first turn after a provider launch. ## Risks - Only local Codex gets the fast path. ACPX streaming keeps the old 250 ms wait after each ACK. A later change can opt ACPX in when its poll can prove that it does not restore the sidecar. - More provider polls during Codex streaming: a poll can now follow each ACK at once instead of after 250 ms. Batches can become smaller, so the provider state save for each batch can happen more often. Each event already has its own durable save. - A command that the controller sends more than 1 ms after its ACK can now arrive after a provider poll. This was already possible after 250 ms. The change applies only when this connection produced the output, so the provider is already active. - If the Runner cannot restore the 250 ms read timeout, it treats the socket as failed and reconnects. It does not continue with a 1 ms timeout. - No protocol, schema, or controller change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, Honeycomb queries) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
199670ba99 |
fix(accounting): prevent usage receipt stalls during streamed run updates (#20385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner commits usage receipts before it acknowledges retained events. > - Usage receipts and streamed run logs update the same run row. > - A repeatable-read snapshot can become stale before the receipt writer locks that row. > - Continuous log writes can exhaust all accounting retries and stall the retained event. > - This pull request uses the existing run lock with READ COMMITTED for standalone receipt writes. > - Other accounting operations keep their consistent snapshot. Stall diagnostics retain safe nested error codes. ## Linked Issues or Issue Description Related: #20235. **What happened?** A hosted native run stopped on a retained usage event. Its diagnostic reported `Error at source seq N`. The saved outbox and accounting records locate the failing delivery boundary. Traces show repeated accounting transactions while streamed events still updated the run row. The historical inner exception is unavailable. A controlled PostgreSQL reproduction proves a matching defect: a run-log append after company validation makes the receipt writer's repeatable-read snapshot stale. Repeated appends exhaust all four retries with SQLSTATE `40001`. This is a demonstrated failure mechanism; attribution of the historical incident remains an inference. **Expected behavior** Commit the receipt under its existing run-row lock despite concurrent run-log writes. Observe concurrent finalization and preserve late-receipt debt. Keep duplicate receipts idempotent and reject conflicting or cross-company receipts. **Steps to reproduce** Run `server/src/services/usage-receipts-read.test.ts`. The concurrency regression uses real PostgreSQL and the production run-log append function. It commits a log append after company validation and before the receipt locks the run. Before the repair, the receipt rejects with a wrapped `40001` after four attempts. After the repair, it commits once. The finalization variant verifies that a newly settled run receives late-receipt debt without replacing settled state. ## What Changed - Allow aggregate-lock-protected accounting writers to select READ COMMITTED. Keep repeatable read as the default. - Use READ COMMITTED only for standalone usage receipt writes. Keep the existing run lock, immutable receipt journal, replay checks and transactional budget-work publication. - Add PostgreSQL regressions for concurrent log appends, concurrent finalization, duplicate delivery and the unchanged default snapshot. - Read the outer event-commit error and at most three nested causes for an existing safe error code. Exclude exception messages and SQL. - Document receipt isolation and bounded stall diagnostics. ## Verification - Red: the production receipt writer exhausted four retries with PostgreSQL `40001` under controlled concurrent run-log appends. - Red: wrapped database and transport stall diagnostics returned `Error` instead of `40001` and `ECONNRESET`. - Green: Runner control-plane tests passed all 207 cases. All 68 server assertions passed across receipt reads, native usage delivery and cost accounting. Cost-accounting teardown exceeded its 30-second limit twice on this loaded host, after all 28 assertions passed. The workspace project retained its configured 30-second hook limit despite the CLI override. This is a local suite cleanup failure, not a passing suite; the same revision passed the complete server suites in CI. - Server layered TypeScript checking passed, including source and test layers. Runner TypeScript checking passed. Server checks cover `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`; the unchanged Runner checks cover its parent. - Local environment: macOS arm64, Node 26.4.0, disposable embedded PostgreSQL. Native usage-delivery tests use the real Rust receiver. - CI at `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`: [build](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568428), [typecheck](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568367), native Runner integration, browser shards and all nine serialized server groups passed. All 12 general server shards passed in the [full run](https://github.com/paperclipai/paperclip/actions/runs/42714760702). Final rollup: 59 successful checks/statuses, three intentionally skipped checks, no pending or failed checks. - [Greptile reviewed this revision at 5/5 with no actionable findings](https://github.com/paperclipai/paperclip/pull/20385#issuecomment-6753163397). - Hosted evidence came from the task, run, saved sandbox outbox, accounting receipts and traces. The run started fresh and did not cross a deployment. No original agent turn was replayed. - The hosted repair has not been deployed. The original task has not been resumed. A fixed-revision remote-provider user journey remains unverified. ## Risks - READ COMMITTED changes snapshot timing for standalone receipt writes. The run-row lock protects all mutable receipt/accounting reads and serializes competing receipts and finalization. Other accounting operations retain repeatable read. - Receipt identity, replay conflict checks, company scoping, budget publication, retry bounds and acknowledgement rules remain intact. - No persisted format, provider identity or database schema changes. Existing saved state needs no migration. - Diagnostics inspect at most four objects and retain only the existing identifier format, with an 80-character code limit. - The reproduced race matches the incident boundary and timing, but the original inner exception was not recorded. Rollout and live acceptance remain necessary before incident resolution. ## Model Used - OpenAI Codex, GPT-6 family, with code and shell tools. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
0e325e8d52 |
fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
79bf733012 |
fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner keeps a lifetime fence until the original ACPX provider stops. > - A recovered Runner must prove that fence is free before it clears saved cleanup state. > - Suspension and other recovery paths already wait for transient listener holders. Recovered shutdown used one immediate probe. > - That difference made parallel Rust tests fail after they dropped their original listeners. > - This pull request uses the existing bounded proof for recovered shutdown and removes the repeated test retry. > - The benefit is one cleanup rule, with the same fail-closed proof and no new work on normal task dispatch. ## Linked Issues or Issue Description Fixes: #20210 Thanks to @genius-0963 for the test-only fix proposed in #20321. That related alternative adds a test-only shutdown retry. This change applies the existing bounded cleanup proof in the shared shutdown path and removes the same retry pattern from native-provider integration tests. It also tests a single shutdown call while the original quorum is still held. ## What Changed - Use `await_provider_lifetime_fence` for a recovered executor with no live session and unconfirmed provider exit. Delete the unused one-shot helper. - Keep the acquired quorum held through the existing durable state write. A failed proof leaves `providerExitUnconfirmed` set. - Give the shared timeout message wording that applies to shutdown and suspension. - Extend the recovery test with a worker-start signal. Hold the original listeners until the single shutdown call is waiting, then release them and assert successful cleanup without a provider session. - Use the same bounded proof in the retired-quorum fixture test. Remove the duplicate retry loop from the native-provider test helper. - Document the bounded recovery cleanup wait. ## Verification - macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only targeted checks ran locally. - `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`: passed. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on the rebased head (5.21 seconds). The initial head also passed five more parallel-test runs (45 tests per run), with no failure. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test native_provider_backend acpx_`: 14 passed on the rebased head (9.43 seconds). These tests use the real Rust executor and controlled sidecars. - Negative control: restore only shutdown's original one-shot proof. The new regression fails at `shutdown must wait while the original lifetime still holds a quorum`. Restore the fix and the affected module passes. - Independent parent verification on current commit `368fc814f7`: the three exact recovery, held-quorum, and retired-quorum tests passed (3 tests, 5.06 seconds). Independent source review confirmed the unchanged three-file scope and found no actionable issue. - The active-turn recovery test still holds a live quorum through the five-second bound. It checks that shutdown refuses cleanup, saved state remains unconfirmed, and the provider never starts. It then releases the quorum and confirms cleanup. - Standard Greptile scored current commit `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads. - Initial GitHub general-server shard 7 failed in unchanged `remote-workspace-nested-verification.test.ts`, at its 15-second wait for a child-process import checkpoint (expected one row, got zero; both Vitest attempts). [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815). That fixture runs Node, PostgreSQL, Git, and storage migration; it does not run the native Runner or the changed Rust cleanup. The cause of the checkpoint timing miss remains unknown. The focused retry passed on the unchanged source ([successful retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)). This proves recovery, not the cause of the original timing miss. GitHub kept the reusable-workflow aggregate failed after that job-only retry and rejected both a final-gate retry and a failed-job retry. A subsequent run-level retry materialized only the gate and aggregate jobs; its aggregate inputs reported prior jobs as `abandoned`, so it failed without a new test failure. A new `full-ci` label event requests a fresh workflow graph on the same commit. The reason for those rerun-state results is not established. - The fresh `full-ci` run [42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208) then lost several runners mid-job. The maintained [runner-loss recovery 42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525) classified all four failed jobs as runner loss: typecheck, E2E shards 2 and 8, and server shard 4. The evidence was a shutdown log or lost-communication annotation. It retried the failed jobs together; canceled sibling jobs were also requeued. This is runner-interruption evidence, separate from the earlier checkpoint assertion. - The maintained recovery attempt again failed the same partial-checkpoint assertion in unchanged `remote-workspace-nested-verification.test.ts`, twice ([job 129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)). Neither checkpoint-failure job logged runner shutdown or lost communication; their resource summaries reported no significant constraint. The child phase at the timeout remains unknown. No further blind retry or TypeScript change is included here. Related startup-handshake work exists in #20339; its controlled-delay evidence establishes a deadline-boundary defect but does not establish the phase of these original CI failures. - Rebased onto current master after upstream #20339 merged. That separate change fixes a demonstrated child-startup/checkpoint-deadline boundary. It does not prove which child phase caused the earlier zero-checkpoint assertions. This PR still changes only its original Rust and documentation files. Fresh GitHub CI on the new base will establish recovery, not retrospective cause. - Full local tests and a local release build were not run. Current-head [PR CI 42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411), [Runner Windows 42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803), and [Docker Runner 42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601) all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check list has 59 successes and 5 intentional skips. GitHub supplied broad typecheck, build, and optimized Rust coverage. This is new-base recovery; the cause of the original checkpoint assertion failures remains unproved. ## Risks - A genuinely unconfirmed original provider can now hold recovered shutdown for up to the existing five-second proof deadline. The proof returns immediately when two candidate ports are free. - The changed branch requires both no in-memory provider session and unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch keep their existing paths. Outer controller close deadlines remain authoritative. - State saved by the previous release carries forward unchanged. No schema, launch identity, profile digest, protocol, or retry-budget change. - No live provider session or hosted deployment was tested. The tests cover cleanup proof, refusal, durable state, and controlled provider execution. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
4a349ea447 |
fix(chat): tell agents that instruction saves keep no history (#20328)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents can read and edit other agents' instruction files with dedicated tools > - #14420 changed instruction saves: Paperclip now keeps only the current file and records no revision history > - The agent-facing descriptions still describe a "canonical instruction revision" history > - In a live eval, a chat agent asked to undo an instruction change found an empty history, read it as a test artifact, guessed the earlier text from the current file's wording, wrote it, and reported a restore > - This pull request tells agents the real contract in chat guidance and in the tool and API descriptions > - The benefit is that an undo with no saved version leaves the instructions alone and asks the person, instead of inventing an earlier version ## Linked Issues or Issue Description Refs #14420 Refs #19515 (open: would list save records from the activity log. Those records hold no content, so they still cannot restore earlier text) **What happened** A person asked a chat agent: "Undo Morgan's last instruction change." Morgan's history was empty, because saves do not record revisions. The agent's reasoning said the history "is only populated with canonical revisions" and that there was "no revision tracking in this e2e harness". The agent then removed the last line of AGENTS.md and replied "Undone. Morgan's AGENTS.md is back to …". No saved source had the earlier text, so the agent guessed it. **Expected behavior** When there is no saved earlier version, the agent leaves the instructions unchanged. It says plainly that no earlier version was saved, and it asks the person for the text they want or which part to remove. **Steps to reproduce** 1. Create an agent with an AGENTS.md file. Edit the file once through `PUT /api/agents/{id}/instructions-bundle/file`. 2. `GET /api/agents/{id}/instructions-bundle/history?path=AGENTS.md` returns `{"revisions":[]}`. 3. Ask a chat agent to undo the agent's last instruction change. The agent rewrites the file from a guess. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says three things: - Saves keep only the current instruction files. - An empty history means no earlier text is retained. - An undo with no listed revision must leave the file unchanged, say that no earlier version was saved, and ask for the wanted text. - The `get_agent_instruction_history` contract now says that it lists only pre-upgrade revisions, and that an empty list means Paperclip retains no earlier version. - The `restore_agent_instructions` contract now says that only listed pre-upgrade revisions can be restored. - The OpenAPI summary for `GET /api/agents/{id}/instructions-bundle/history` says the same. - The semantic tool contracts, the protocol manifest and the seeded eval catalog hash are regenerated. - Regression tests check the new guidance and both live tool contracts. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts server/src/__tests__/board-access-denial.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/runner-api.test.ts` passes. - In `packages/paperclip-runner`, `pnpm exec vitest run src/semantic-tools/paperclip-semantic-tools.test.ts src/catalog src/evals/native-execution.test.ts` passes. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-protocol-manifest.mjs --check` and `generate-replay-goldens.mjs --check` pass. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - The live result is the chat eval `agents.restore-instructions`, run with this change cherry-picked: run 42666621831 passed. The agent replied that Morgan's history is empty and no earlier version exists. It left AGENTS.md unchanged and asked for the exact text or the line to remove. Run 42660236086, without this change, failed: the agent guessed and rewrote the file. ## Risks - Low risk. Only agent-facing text changes, and no behavior changes. - The runner semantic catalog hash changes. This is the same kind of change as #19983. - If #19515 lands, history lists save records without content. The guidance wording must then say that listed saves without content cannot be restored. Whichever PR merges second must reconcile the two texts. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
76254ead22 |
fix(runner): allow Codex interruption to settle governed waits (#20330)
## Thinking Path > - Paperclip manages AI agents and their work. > - A native run must stop provider work before it waits for a human response. > - The Codex transport allows 30 seconds for a control command. > - The governed-wait loop used its 15-second default for Codex. > - A valid interruption terminal at 18 seconds therefore failed the run. > - This change gives Codex the existing 30-second terminal drain allowance. > - The run still fails if it cannot prove that the provider turn ended. ## Linked Issues or Issue Description **What happened?** A Codex run can save a human wait, accept an interruption, and fail before the provider sends its terminal. Its saved wait remains intact, but normal finalization does not occur. **Expected behavior:** Allow terminal delivery through the transport's bounded command window. Keep missing-terminal cases closed. **Steps to reproduce:** Run the new protocol regression against the old backend. The real Runner binary uses a controlled Codex process that delays its interruption terminal for 18 seconds. The run fails at the 15-second drain deadline. The same process passes with this change. **Paperclip version or commit:** Reproduced against `1b7fe33432`. **Deployment mode:** Local test of the native Runner path. The incident was observed in a hosted sandbox. **Installation method:** Source checkout. **Operating system:** macOS arm64 for local verification. Hosted incident runtime was Linux. **Additional context:** No matching repair was found. This addresses a reproducible deadline mismatch. The hosted log ends terminal observation at 15 seconds, so it cannot prove whether that provider would have settled within 30 seconds. The approved local live-provider check passed. This PR does not claim recovery of the hosted task. ## What Changed - Give Codex a 30-second governed-wait terminal drain through the existing backend option. - Keep other providers and the caller's shorter execution bound unchanged. - Add real Runner protocol cases for prompt, delayed, and absent terminal delivery. Check retained wait state and one provider turn. - Document the bounded wait and terminal-proof requirement. ## Verification - Red: the 18-second controlled-provider case failed with `governed_wait_provider_terminal_unproven` on the old backend. The prompt case passed. - Green: 282 tests passed across the native runtime, harness backend, and new Runner protocol suite, including a repeat on the rebased PR revision. `pnpm verify --explain` returned exit 2 because the execution-contract document requires broad CI; the [required CI run](https://github.com/paperclipai/paperclip/actions/runs/42668235142) passed on the final head. All 58 applicable checks passed; the three skipped checks were the two unrelated Storybook jobs and the selected-test planner replaced by broad coverage. - The missing-terminal case still fails without `completeRun` or a successful run terminal. The saved governed wait remains available. Only one provider turn and one tool call occur. - Existing runtime tests cover shorter deadlines, foreign or unproven terminals, complete usage without a terminal, and checkpoint recovery without another model turn. - Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64, disposable local workspace. One `request_human_input` tool call returned a pending interaction fixture. The real provider emitted `turn.interrupted`; the controller accepted the yielded wait and its run terminal. The check passed in 8.14 seconds. The tool callback and persistence port were controlled; no hosted server permission or Notebook publication claim is made. - Initial local probe setup failed before model execution: first the explicit launch lacked `app-server`, then it omitted the Runner permission configuration. The successful check used the Runner's normal launch arguments and isolated home with the approved existing account. - Hosted sandbox acceptance and recovery remain unverified. The 18-second delayed shutdown is a controlled protocol regression; the live model check proves normal shutdown, not that specific latency. ## Risks - A Codex provider that never stops can occupy the settlement window for up to 15 additional seconds. It still fails closed and is quarantined. - No persisted schema, provider identity, permission, or recovery rule changes. Previous-release checkpoints use the same format and existing recovery checks. - This change cannot fix a provider that never emits a terminal. It also does not grant missing setup permissions or decide pending approvals. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool calls, code edits, and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
97188f276c |
perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.
## Linked Issues or Issue Description
Refs #20190, #20285
**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).
**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.
**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.
## What Changed
- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
- New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
- The other `DurableRunnerConfig` literals set `turn_scoped: false`.
## Verification
- `cargo test -p paperclip-runner-core`:
- `--lib`: 602 passed.
- `--test acpx_park_preparation`: 22 passed.
- `acpx_park_readiness`: 8 passed.
- `acpx_warm_attachment`: 4 passed.
- `acpx_provider_suspend`: 5 passed.
- `acpx_provider_session`: 12 passed.
- `acpx_provider_turns`: 29 passed.
- `native_provider_backend`: 47 passed.
- `durable_recovery`: 3 passed.
- `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(
|
||
|
|
fd2e7482c0 |
fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop
## Linked Issues or Issue Description
Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).
**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.
**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.
**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.
**Paperclip version or commit**
master at
|
||
|
|
e015c2ef21 |
fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path - #20145 prepares the next Claude provider generation at park, and #20190 restarts the ACPX provider when a turn settles. They move the sidecar respawn off the follow-up's critical path. - When that turn-end restart fails, the runner "fails closed": `restart_session_after_turn` saves the session `closed` and drops it. Within the run, the next turn fails exactly like a failed restart at turn start. That's intended. - Across runs it breaks the conversation: - the next run's `run.attach` refuses a `closed` session ("run.attach requires the same settled ACPX provider profile and session"); - it reconnects for about 10 s; - the controller then records a continuity break and starts a fresh Claude conversation (`run.startup.reuse provider_session: fresh/created`). - A local A/B (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`, on a host with load average about 17–27) confirmed it: | | |
||
|
|
09412e652a |
fix(runner): allow warm release to drain after provider restart (#20290)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm Runner sessions park after a turn so follow-up tasks can claim the same process. > - Claude restarts its ACPX provider after the controller receives the terminal result. > - That restart can delay the drain receipt beyond the release path's one-second limit. > - A failed drain prevents parking, so the idle timer later closes the retained Runner. > - This pull request uses the existing warm admission budget for release drain. > - The benefit is retained Runner reuse without moving provider restart onto task dispatch. ## Linked Issues or Issue Description Fixes: #20225. Refs #20190 and #20145. Related performance follow-up: #20224. ## What Changed - Align the release drain timeout with the existing 30-second warm admission budget for every provider. Close paths keep their explicit remaining deadlines. - Keep the existing immediate completion when the drain receipt proves settlement. Add no sleep, retry, provider branch or dispatch work. - Add a real Runner regression with a fake ACPX sidecar whose replacement start takes 1.5 seconds. One release call must park the Runner, then a follow-up must claim the same Runner process. - Document the release drain budget. ## Verification - Targeted command in `packages/paperclip-runner`: `vitest run src/live/runnerd-codex-transport-release-resume.test.ts src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed** across 3 files, 157.78 seconds. - Independent parent verification reran the slow-restart and ordinary cached cases: **2 passed**. Source review confirmed that release is the only caller using the changed default; close callers supply explicit deadlines. - Negative control: with the original one-second timeout, the new test fails because the single release call returns `false`. The fixed code passes the new case. - Built only the current-source Runner and two fixture binaries required by these tests. The first sandboxed invocation could not bind its loopback listener (`EPERM`); the test ran with loopback access afterward. - Full GitHub PR workflow [42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993) passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard Greptile scored this same head 5/5 with no review threads. Full repository checks ran in GitHub CI, per the requested local test scope. No live Claude session or hosted deployment was tested. ## Risks - An unresponsive Runner can now hold release drain for up to the existing 30-second warm admission limit. A healthy Runner returns as soon as its receipt arrives. The provider restart stays at turn end; task dispatch gets no fixed delay. - Saved state, protocol and provider identity formats do not change. Previous-release state carries forward under the existing validation. - The controlled fake-sidecar test proves the restart timing race and process reuse. It does not qualify live provider latency or a hosted rollout. ## Model Used - OpenAI GPT-6 through Codex, with repository tools, code editing and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3374ca54f5 |
fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8eb11fa8aa |
fix(runner): fail closed with a recorded cause when runner event commits stall (#20235)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Runner streams numbered events to the control plane, which commits each one and acknowledges the committed prefix > - In restart chaos wave `w-20261109T093417-s9807`, t01 (claude R1, local), the control plane never committed Runner seq 34. Every failure path in `#event` closed the connection without recording why. The Runner reconnected 2,848 times over 14 minutes and nothing detected it > - This pull request makes that loop a detected, named failure that carries its cause > - The benefit is that a wedged event stream becomes a visible run failure with a reason, instead of a silent "running" task ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T093417-s9807` (master `df4562ab38`), task t01, run `d08dc899`. **What happened** (Phase 2's re-check of the kept instance): - The provider kept working. Claude's transcript runs to 09:47:03 and ends in `paperclip_finish`. - The Runner, its ACPX provider state, and the controller journal agree on acked seq 33. The controller never commits Runner seq 34 (Write progressed, 09:36:25.892). - Phase 2 replayed the kept outbox offline. Runner seq 34 fails `validatePrpEvent`: its `/payload/target` contains a newline from a malformed Claude Write call. `#event` closes without a reason, and the Runner redelivers the same bytes forever. (Phase 2's separate PR stops the producers from emitting such paths.) - Every `#event` failure path closed the connection without recording a reason, so the cause was invisible. An outbox that already holds such an event can only be cleared by failing closed. - The `<instance>:<run>:N` ids in `heartbeat_run_events` come from the TS driver's own projection numbering, so there was no Runner sequence rewind. ## What Changed - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: - `#eventCommitFailed` replaces the bare `connection.close()` on each `#event` commit-failure path. It records a bounded cause: `event_sequence_mismatch` for an out-of-order event, `event_schema_invalid:<issue paths>` for an event that fails `validatePrpEvent`, the error's code or name (never its message) for a commit callback error, `journal_sequence_not_contiguous`, `journal_commit_refused`, or `checkpoint_flush_failed:<cause>`. - If failures continue for the same run at the same acknowledged cursor for at least 20 attempts and 45 s, it fails closed with `NativeSessionProtocolIntegrityError("runner_event_commit_stalled", "<cause> at source seq N")`, where N is the failing event's own sequence. Only a moved acknowledged cursor counts as progress (a replayed, already acknowledged duplicate does not), and a different run starts a fresh counter. A single transient failure still just reconnects, as before. - A checkpoint flush that fails before any outbound frame also counts, as `checkpoint_flush_failed:<cause>`. That includes the authentication challenge and the welcome, so a reconnect that cannot even authenticate still reaches the bound. - A journal write that throws (`#store.commit`, `#store.commitEvent`, or the welcome save) leaves the store unwritable until reload, so it fails closed at once with `journal_write_failed:<code>`. - `packages/paperclip-runner/src/contracts/native-session-backend.ts`: new reason `runner_event_commit_stalled`, with an optional `detail` in the message. ## Verification Red/green: - "bounds commit-stall reconnects for an out-of-order event": a schema-valid event with the wrong `sourceSeq` fails closed with `event_sequence_mismatch at source seq 5`. Before this it closed silently forever. "… for 20 failures inside 45 s" pins the time bound: 20 failures over 19 s do not fail closed. - "fails closed when checkpoint storage keeps failing before a reconnect authenticates": with checkpoint persistence refusing (`ECONNREFUSED`), 20 reconnect attempts fail before authentication, then the control plane fails closed with `checkpoint_flush_failed:ECONNREFUSED at source seq 1`. Without this, those attempts close silently. - "fails closed at once when the journal cannot record a runner event": a journal write throwing `ENOSPC` fails closed with `journal_write_failed:ENOSPC at source seq 1`, and the private error text is not carried. On master the connection only closes. - `durable-prp-control-plane.test.ts` "fails closed with the schema issue when the runner keeps redelivering an invalid event (restart chaos t01)": an event that fails `validatePrpEvent` is redelivered 20 times over 57 s of mocked clock. On master the control plane only closes; with this PR it fails closed with `runner_event_commit_stalled` and detail `event_schema_invalid:/… at source seq 1`. - `durable-prp-control-plane.test.ts` "fails closed when runner events keep failing to commit without progress": 20 reconnects over 57 s of mocked clock, each commit failing with `ECONNRESET`. On master nothing fails closed. With this PR the 20th failure reports `runner_event_commit_stalled` with detail `ECONNRESET at source seq 1`, and the private error message is not carried. File: 201 passed, including the existing "keeps a non-typed commit failure retryable" cases. Static checks: - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. The server is unchanged. ## Risks - Storage that keeps failing event commits for at least 45 s and 20 attempts now fails the run (operator recovery) instead of looping silently. Shorter outages are unchanged. - The recorded cause is a code or error name only, never error text, so no tool content or credentials reach the run error. - No schema change and no change to persisted state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3f2a940d5b |
fix(runner): answer held Paperclip tool calls after a local restart before the next turn (#20227)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart interrupts a turn, the Runner keeps each Paperclip tool call the model made as "held" until Paperclip answers it, and it refuses to start the next turn while any call is held > - Paperclip answers held calls after a reattach through its tool recovery path. That path was wired only for remote runners > - In restart chaos wave `w-20261109T084023-s9403`, t101 (local codex, K2 then K1) held a `read_document` call that nothing answered, so the continuation's `turn.start` failed and the task needed a Retry > - This pull request wires held-call recovery for local runners too, and makes `turn.start` answer or wait for held calls first > - The benefit is that a turn lost to a restart no longer strands the run behind one unanswered call ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T084023-s9403` (master `fc5c79a327`), task t101 (codex C1, K2, local). **What happened:** - Run `4b5810d5` called `read_document` (`exec-ac8fa9b4`) around the K2 drain at 09:08:18. A K1 then killed the next server start. - At 09:08:36 the third controller reattached. The Runner restored the conversation and reported the old turn lost (`provider_turn_lost_on_restore`, recoverable). The held call stayed pending, by design: a held call accepts a late result. - The controller had the call: the quarantine detail lists `exec-ac8fa9b4` under `semanticTools.pending`. But `recoverPendingSemanticInputs()` returned at once: for a local run `controllerPersistence` is unset, so `recoverSemanticToolInput` was `undefined`. - At 09:08:46 the continuation's `turn.start` failed with "cannot prepare the next provider turn while tool calls are active (pending: 1 [read_document:exec-ac8fa9b4…])". The run ended `native_session_interrupted`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: the runnerd transport now always gets `recoverSemanticToolInput`, not only when `controllerPersistence` (remote) is set. It uses the same epoch check and `SessionAuthorityEpoch.recover`. Recovery re-authorizes and re-runs reads, and replays receipts for mutations; it never repeats an uncertain mutation. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts` (`#startTurnWork`): when the control plane has unsettled semantic inputs, it runs `recoverPendingSemanticInputs()` before queueing `turn.start`, and waits for that pass, bounded by the turn-start deadline. Recovered results are queued as commands ahead of `turn.start`, and the Runner applies commands in order. If recovery has not finished by the deadline, the turn start fails with `native_turn_start_held_tool_calls_unresolved` instead of sending `turn.start` behind it. When everything is settled there is no wait, so a normal turn start is unchanged. Live calls of a turn being replaced are not waited on; the replaced start still rejects them. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: `recoverPendingSemanticInputs()` passes are serialized. A call made while the startup pass is still answering a held call waits for that pass, then runs its own, so the turn-start gate cannot finish early by skipping a call that is already being recovered. ## Verification Red/green: - `native-session-executor.test.ts` "answers a local run's held Paperclip tool calls after a restart (restart chaos t101)". On master `recoverSemanticToolInput` is `undefined` for a local run; with this PR it is wired. - `runnerd-codex-transport.test.ts` "answers a held call before starting the next turn (restart chaos t101)", against real runnerd and fake Codex, with a slow startup recovery pass already in flight when `turn/start` arrives. On master `turn/start` never ran recovery, and without the serialization `turn.start` is queued before the held call's result. With this PR the held call is recovered once and its result is queued before `turn.start`. Regression and static checks: - `runnerd-codex-transport.test.ts`, `durable-prp-control-plane.test.ts`, `runnerd-codex-transport-adoption.test.ts`: 510 passed. - `native-session-executor.test.ts` and `native-session-resume.test.ts`: 1368 passed, 1 failed. The failing case, "archives a damaged prior epoch … with real runnerd", fails identically on master on this Mac (`ENOENT … authority-epochs`), so CI is its reference. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Local runs now recover held calls after a reattach, as remote runs already did. Reads re-run under current authority. A mutation without a saved receipt still returns no result and stays held, as before. - `turn.start` may now wait up to its existing deadline while a recovery pass runs, then fails clearly if it has not finished. It never waits when nothing is unsettled, so normal turns get no added latency before `turn.start`. - No schema change and no change to persisted run state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
465da4227a |
fix(runner): drop a tool target path that breaks a line so the controller can acknowledge its event (#20238)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner turns provider tool calls into durable PRP
events. The controller acknowledges each event only after it validates
and commits it.
> - The controller's event schema forbids line terminators in a safe
path such as `tool.execution` `target`.
> - The ACPX sidecar and the Rust runner accepted a location path that
contained a newline. Claude sometimes emits a malformed Write call that
puts the file content into the path.
> - The controller refused that event without an acknowledgement. The
runner redelivered the same bytes, and the run stopped making progress
for 14 minutes (2,848 reconnects).
> - This pull request makes both producers drop such a path, so the
event carries a null target and the controller accepts it.
> - The benefit is that a malformed provider path can no longer wedge a
run's event delivery.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9807 (claude, R1, local), the run stopped
receiving events after runner source seq 33. The runner reconnected
2,848 times in 14 minutes. Offline replay of the saved runner outbox
through the controller's `validatePrpEvent` showed that seq 34 fails
`/payload/target` (the safePath pattern), because the target is
`packages/shared/src/chaos/slug-2c5c3c8f.ts\n<parameter
name="content">export function chaosSlug…`. Seq 35, 39 and 40 (the same
tool call) fail the same way.
**Expected behavior**
The runner never emits an event that the controller's schema refuses. A
malformed tool path becomes a null target, and the tool event still
arrives.
**Steps to reproduce**
1. Have an ACPX provider send a tool call whose location path contains a
newline (for example, a Write call with the content in the path).
2. See that the runner emits `tool.execution.progressed` with that
target.
3. See that the controller closes the connection without an
acknowledgement on every redelivery.
**Paperclip version or commit**
master at
|
||
|
|
d7d0f96f3f |
perf(runner): save provider state once per poll instead of once per frame (#20217)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode, Codex and Cursor Cloud chat turns run through runnerd. runnerd reads provider frames in order and keeps the run's provider state in a durable file > - For every frame that it reads, runnerd rewrote the whole provider state file and called `fsync` on the file and on its directory. The state holds every tool result kept in the turn, so each save got more expensive as the turn went on > - A tool call request is just the next frame. It waited behind one full save for every frame before it. #20209 measured tool-call turnaround that grew from about 1.4 s to about 15 s in one turn > - #20209 stopped the OpenCode proxy from sending frames that runnerd discards. Every frame that runnerd still reads still paid a full save, and other proxies (for example Cursor Cloud) still send such frames > - This pull request makes runnerd save the frames of one poll once, before anything derived from them leaves runnerd. A save that would write the bytes already on disk is skipped > - The benefit is that the number of saves before a tool call no longer grows with the number of streamed frames. In a local measurement, a tool call after 400 frames with 830 KB of kept state took 0.57 s instead of more than 30 s ## Linked Issues or Issue Description Refs #20209 Related: #17514 (the same per-event full-state rewrite, on the server control-plane side) **What happened** During a native OpenCode chat turn, tool-call turnaround grew from about 1.4 s to about 15 s as the turn went on. runnerd saved its complete durable provider state, with `fsync`, after every provider frame. That state includes the tool results kept so far in the turn. A tool call request waited behind one full save for each earlier frame. **Expected behavior** The time to deliver a tool call must not grow with the number of assistant-text, reasoning or item frames that came before it. A crash or restart must still never lose state that a resumed run needs. **Steps to reproduce** 1. Start a native OpenCode (or Codex app-server or Cursor Cloud) turn that keeps a few tool results and then streams many text or reasoning frames before its next tool call. 2. Measure the time from the tool call frame to the `semantic_tool.input` event. 3. See that the time grows with the number of earlier frames and with the size of the kept state. Locally, `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture` on `master` reports 102 saves and 9.3 s for 100 frames with 830 KB of kept state, and more than 30 s for 400 frames. ## What Changed - `packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs` and `codex_provider.rs`: - The provider frame loop moves into `poll_provider_frames`. An ordinary notification frame and a descendant notification frame no longer save the state. They mark it as unsaved. - After the loop, `poll_current_provider_with_control` saves the unsaved state once. It also saves when a frame fails, before it returns the error. - Frames that already saved at once still do so: turn terminals, Cursor Cloud terminal deliveries, `reconciliation_required`, protocol failures, MCP permission and runtime requests, process exit, and new tool calls. - A tool call that runnerd answers at once (an exact completed replay or the receipt-limit stop) saves unsaved state before it writes to the provider. A rejected tool call already saved first. - `retained_events`, `execute` and `host_park_ready` save unsaved state first, so no event, command result or park decision can expose state that is not on disk. - Courtesy rejections that `CodexProvider::poll` writes by itself (unsupported requests, descendant requests, delayed requests after a terminal, full runtime-request capacity) are held while runnerd reads a batch (`hold_courtesy_responses`). runnerd sends them only after it saves the state that covers every earlier frame (`save_before_provider_writes`). A terminal frame also saves and releases them before runnerd asks the provider for goal state or drains the accounting tail. Released courtesy replies are best-effort, as courtesy replies were before: a provider that already closed stdin cannot turn a received terminal into a polling failure. The receipt-limit stop sends held cancellations before its `turn/interrupt` request, so their order to the provider does not change. - The trace records for runnerd's normalization (`rust_durable_normalization`), which name emitted event ids, are kept until the state that holds those ids is saved. The trace therefore only names saved events. Saved mappings are written before an exited or failed provider is dropped, so they are not lost. Raw frame records still go to the trace when a frame is read; they name no event ids. - `save_unsaved_frame_state` does not write when the serialized state is byte-for-byte the same as the saved file. All other saves write as before. - Serialization moves to `serialize_provider_state`, and the file write moves to `write_state_bytes`. The write path (private temporary file, `fsync`, rename, directory `fsync`, `0600`) does not change. - New tests in `packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs`: - `streamed_frames_do_not_multiply_durable_saves_before_a_tool_call`: 8 and 48 streamed frame pairs before a tool call need the same number of saves (1). On `master` they need 18 and 98. - `batched_frame_state_is_saved_before_events_leave_and_survives_a_restart`: at every poll, the saved file is equal to the in-memory state and holds exactly the delivered events. A new executor loads the file after the old one stops and gets the same state, the same unacknowledged events and the pending tool call. - `batched_frame_state_is_saved_when_a_later_frame_fails`: a frame that fails after 20 batched frames still leaves those 20 frames on disk. - `previous_release_state_is_rewritten_by_the_first_batched_save`: a state file in an older shape (compact JSON without newer default fields) loads, and the first batched save rewrites it. Unchanged state is not rewritten. Changed state is. - `courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved`: when the courtesy rejection reaches the provider, the saved file (what a restarted runnerd would load) already holds the 20 earlier frames and equals the in-memory state. It fails if the hold is removed. - `trace_interpretations_name_only_saved_events`: every event id that a trace record names is in saved state at the moment the record is written. It fails if records are written before the save. - `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`: after 122 frames are read and not yet saved, the saved file is byte-for-byte unchanged and runnerd has written nothing to the provider. A reload gets the pre-batch event sequence and no pending events. - `a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`: the provider closes stdin and then completes the turn with a tool call still pending. The held cancellation cannot be written, and the turn still completes and is saved. - `trace_interpretations_survive_a_provider_exit_in_the_same_poll`: the trace keeps the mappings for frames that arrived in the same poll as the provider exit. - `measure_streamed_frame_cost_before_a_tool_call` (ignored): prints saves and tool-call latency for different frame counts and kept-state sizes. ## Verification - `cargo test --locked -p paperclip-runner-core` (debug, macOS arm64): library 595 passed, 1 failed; all 35 integration test binaries passed (`--no-fail-fast`). The one library failure is `task_memory::tests::monitor_bounds_notices`. It also fails when run alone, takes about 9 minutes, and waits on file-watch timing in `task_memory.rs`, which this PR does not touch. In one earlier run, two ACPX park tests in `acpx_park_preparation` failed. The same binary also failed on unchanged `master` (with a different test), and it passed in the final run. - `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture`, debug build, macOS APFS (`sync_all` is `F_FULLFSYNC`). Each row is one turn: frames streamed, then one tool call. | Kept state | Frames | Saves (master) | Tool-call latency (master) | Saves (this PR) | Tool-call latency (this PR) | | --- | --- | --- | --- | --- | --- | | 4 KB | 100 | 102 | 946 ms | 1 | 18 ms | | 4 KB | 400 | 405 | 3,791 ms | 7 | 70 ms | | 211 KB | 100 | 102 | 3,207 ms | 1 | 37 ms | | 211 KB | 400 | 405 | 12,803 ms | 7 | 193 ms | | 833 KB | 100 | 102 | 9,251 ms | 1 | 93 ms | | 833 KB | 400 | about 405 | more than 30 s (test deadline) | 7 | 567 ms | With 400 frames, the remaining saves are one per poll of up to 128 frames and one per controller acknowledgement. - With the debug `paperclip-runnerd` and `fake-codex-app-server` binaries from this branch: `npx vitest run` on `src/live/runnerd-opencode-streaming-tools`, `runnerd-opencode-assigned-tools`, `runnerd-codex-transport`, `runnerd-codex-transport-settlement`, `runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`, `runnerd-saved-continuation`, `runnerd-detach-durability`, and `src/cli/opencode-proxy-output`, `opencode-proxy-events`, `opencode-app-server-proxy`: 11 files, 353 tests passed. ## Risks - Durability. The rule is: nothing derived from a frame leaves runnerd before the state that holds that frame is on disk. That covers delivered events, command results, park decisions, tool results and rejections, courtesy rejections, Cursor Cloud terminal acknowledgements, and trace records that name event ids. The tests above check each of these. - The unsaved window is at most one poll (128 frames). Every frame in it is undelivered: no event from it reached the controller, and runnerd wrote nothing to the provider while it was unsaved. `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged` proves this. A crash in the window therefore leaves the saved state and everything the controller and provider have seen exactly as a crash just before those frames were read. The resumed run starts from the same event sequence, so it cannot deliver an event id twice. Its normal recovery (thread resume and `thread/read` reconciliation) handles those frames in the same way as frames still in the provider pipe. On `master`, a crash could already lose the one frame being processed. - One provider write can still happen in the window: `thread/read`, which `CodexProvider::poll` sends to check the lineage of an unknown helper thread. It is a read-only query that waits for its answer, and it changes nothing on the provider side. - Courtesy rejections now reach the provider at the end of the poll instead of during it. The delay is one batch. The provider gets the same replies. - Previous-release story: the saved file shape does not change, and no new field is saved. A file from the previous release loads as before. The skip check compares exact bytes, so an older file is never treated as already saved. The first save after an upgrade rewrites it in the current shape. A deploy that lands mid-turn sees a file that the previous release saved after each frame. That file is valid input for this code. - The ACPX backend has its own frame loop and is not changed here. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file editing) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9c18cf61da |
fix(runner): keep discarded OpenCode token deltas from delaying tool calls (#20209)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode runs use a TypeScript app-server proxy between OpenCode and runnerd > - runnerd reads the proxy's frames in order, only after the controller acknowledges earlier events > - For each frame, runnerd rewrites its whole durable provider state with fsync, including every tool result that the turn retains > - The proxy sent one `item/delta` frame for each reasoning or tool-output token, and runnerd discards `item/delta` > - A Paperclip tool call request therefore waited behind hundreds of discarded frames, and the wait grew with each tool result in the turn > - This pull request stops the proxy from sending the token deltas that runnerd does not consume > - The benefit is that tool calls reach runnerd without that queue, so agent turns finish faster ## Linked Issues or Issue Description **What happened** In the chat eval `plugins.configure-plugin` (runner-opencode, deepseek-v4.1-flash, GitHub Actions run 42574640038), `paperclip_search_api` calls in one turn slowed from about 1.4 s to 9-15 s each. The server handled every call in 50-250 ms (`tool.execution` spans). For one `call_api` call, 6.3 s passed between the server's 403 and the next agent activity. Time to first token was about 4.4 s. The run events show the cause. Events that OpenCode streams reach runnerd in bursts. In one burst, the usage events for three different model requests arrived within 0.5 s, which no model can do. Each burst ends with the next tool call request. 143 of 203 recorded runner-opencode runs that we checked show such bursts, and 108 show a gap of 8 s or more. Many chat eval cases that time out use this runner. The mechanism: 1. OpenCode streams reasoning one token at a time. The proxy forwarded each token as its own `item/delta` frame. 2. runnerd reads proxy frames in order and only after the controller acknowledges earlier events. For each frame it calls `save_state`. That serializes the whole provider state, with the tool results retained in the turn, and fsyncs it. 3. runnerd's Codex facade has no mapping for `item/delta`, so it discards the frame after the save. 4. A tool call request is one more frame in the same ordered stream. It waits until runnerd works through every earlier delta frame. The wait is about (reasoning tokens) x (save cost), and the save cost grows with each retained tool result. **Expected behavior** A Paperclip tool call reaches runnerd soon after OpenCode makes it. Its delay does not grow with the reasoning length or with the number of earlier tool calls in the turn. **Steps to reproduce** 1. Run a native OpenCode chat turn with a reasoning model that makes many sequential Paperclip tool calls, for example `search_api`. 2. Compare each tool's `item.completed` tool_result time with the `tool.execution.completed` time and with the next `item.started` tool_use time in the run events. 3. See that the gap grows during the turn, and that events for several model requests arrive within less than one second. 4. For a local model of the mechanism: run the pinned OpenCode binary with a controlled OpenAI-compatible provider that streams 300 reasoning tokens before each tool call. Pass the driver events through the proxy's mapping and batching into a reader that consumes one frame at a time with a save cost that grows by 4 ms per completed tool result. Before this change, 8 sequential calls waited 1.7, 3.1, 4.2, 5.4, 6.7, 7.9, 9.2 and 10.5 s behind 142-261 queued frames. After this change, they waited 64-93 ms behind 0 frames. ## What Changed - `shouldForwardOpenCodeProxyItem` now takes the event type and payload. For `item.delta`, it forwards only assistant text (`agentMessage` or `text`), which runnerd maps to `item/agentMessage/delta`. - The proxy keeps reasoning and tool-output token deltas. Completed reasoning and tool items still go to runnerd as `item/completed`. Tool progress still goes as `item/updated`. - New regression test: a step with 1,000 streamed reasoning and tool-output tokens puts the same number of frames before the tool call request as a step with 1 token, and no `item/delta` frame is written. - New test: assistant text deltas are still forwarded. Completed reasoning items are still forwarded. ## Verification - `pnpm exec vitest run src/cli/opencode-proxy-events.test.ts src/cli/opencode-proxy-output.test.ts` in `packages/paperclip-runner`: 15 passed. - Red/green: with the predicate change reverted, the two new tests fail. With the change, they pass. - `tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner` has no errors in the changed files. Local errors exist only in `src/eval/workflow-harness.ts`, which needs a built `@paperclipai/paperclip-eval-kernel`. - Local before/after measurement with the real pinned OpenCode binary (`opencode-ai@1.18.34`), described in Steps to reproduce, step 4. - Not run locally: the suites that need the `paperclip-runnerd` debug binary or the bundled provider entrypoints (`runnerd-opencode-streaming-tools`, `runnerd-codex-transport`, `opencode-proxy-thread-read`). They fail the same way on the unchanged base in this worktree because those build outputs are missing. CI covers them. ## Risks - Low risk. The changed frames were discarded by runnerd. No persisted state shape changes, so no previous-release story is needed. - If runnerd later maps a reasoning delta method, the proxy must forward it with batching, as it does for assistant text. The test names this boundary. - This change does not change the per-frame durable save in runnerd. Mapped frames (assistant text, tool items) still pay it. A runnerd change that skips the save when a notification changes no state would protect other proxies too. That is a separate change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, local benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
42bc238aa5 |
fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records it as `provider_turn_lost_on_restore` and continues in a same-conversation continuation turn > - #20118 taught the event consumer to skip a stale restart-lost terminal of another turn once a continuation is running. It applied only when the checkpoint already recorded the interruption > - When the checkpoint instead names the lost turn as active (the drain hit right after the first turn was submitted), the consumer starts the continuation on the first lost terminal. If the same lost terminal is then delivered again, it was taken as the run's terminal and failed the run > - This pull request applies the skip to any recovered run once its continuation has started. For a checkpointed interruption it still waits for the startup decision first > - The benefit is that a turn lost on restore always continues, however often its lost terminal is delivered ## Linked Issues or Issue Description Refs #20118 **What happened?** Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master |
||
|
|
9046884811 |
perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner executes native agent runs. A host can park an idle
Runner (release it and freeze it) between runs, and resume it for the
next run on the same task.
> - A follow-up run that reuses a parked Runner must claim it first. The
claim is a bootstrap ticket, a checkpoint flush, and the plugin resume
that thaws the Runner.
> - Core made the claim only inside `attachRun`, after all of its
per-run preparation. The claim does not depend on that preparation, so
the two ran in series.
> - This pull request starts the claim while Core still prepares the
run. `attachRun` then waits for the claim that is already in progress.
> - The benefit is a shorter start for every follow-up run on a parked
Runner. The model below gives about 285 ms per run at P50.
## Linked Issues or Issue Description
No public issue exists. The issue is described inline.
**What happened?**
A follow-up run that reuses a parked Runner claims the Runner only in
`attachRun`. Core calls `attachRun` after backend setup: runner
environment, assigned MCP tools, tool authority and the plugin task
owner. The claim waits for all of this, but it uses none of it.
**Expected behavior**
The claim runs at the same time as backend setup. Everything that needs
the new run's tool authority stays at attach.
**Steps to reproduce**
1. Run a Codex or Claude task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. Before this change, the plugin resume starts only
after backend setup ends.
**Paperclip version or commit**
`master` at
|
||
|
|
92a18bb0f3 |
perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner drives Claude through the ACPX sidecar. The sidecar's MCP endpoint lasts for the whole session, so it cannot tell which turn sent a late tool callback. > - Because of this, the Runner reaps the sidecar and provider of a generation that ran a turn before it releases call-ID tombstones and rotates receipt epochs. Today this restart runs at the start of the next turn. > - Every follow-up turn in a run pays that restart on its critical path: about 1.5 s with real Claude. #20145 moves the restart to park, but a second turn in the same run, after the first turn that follows an attach, still restarts cold. > - The restart does not have to wait for the next turn. When a turn settles and the host has its terminal result, the Runner is idle. > - This pull request restarts the provider at that point. The next turn, or the next park, then finds a fresh generation. > - The benefit: same-run follow-up turns start without a restart, park does no restart after a settled turn, and the late-callback guarantee is unchanged. ## Linked Issues or Issue Description Refs #20145, which moved the restart to park. This pull request builds on it. No public issue exists, so the problem is described here. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** `AcpxProviderSession::start_turn_timed` restarts the provider before any turn whose generation already ran a turn. A follow-up turn in the same run waits for `session.suspend`, sidecar shutdown, a new sidecar, `initialize`, `session.open` and `run.attach` before `turn.start`. Since #20145, park does the restart instead, but only for the first turn after an attach. **Proposed behavior** When a turn settles and the controller has ACKed every committed event, the Runner restarts the provider. The next `turn.start` uses the fresh generation and only rotates receipts. Park reuses the fresh generation and does no second restart. **Reason and benefit** The restart leaves the turn's critical path, both for same-run follow-ups and for the turns after a park. The order of reap and rotation stays the same, so a late callback from the reaped generation still cannot reach a later turn. ## What Changed - **Runner loop** (`durable/runner.rs`): new `CommandExecutor::prepare_next_turn` hook, a no-op by default. The runner calls it after the idle provider poll, only when no control frame is waiting and the controller has ACKed every committed event (`acked_source_seq >= highest_source_seq`). So the restart always runs after the terminal result reaches the host. - **ACPX turn-end restart** (`acpx_park_preparation.rs`, `acpx_provider_generation.rs`): - `restart_session_after_turn` restarts a Claude session that has a completed turn, nothing pending, no buffered events, a dormant goal and its whole event tail ACKed. It first settles an abandoned park observation. If that settlement reads more events, it skips this round, and the next idle round restarts after those events are ACKed. - `AcpxProviderSession::restart_after_turn` does the same suspend → reap → start → bootstrap → identity check as the turn-start restart. It records the reaped process id in `restarted_after_turn_from`. This field is the explicit "fresh because of a turn-end restart" marker. - The session does not poll a fresh generation until its next turn or rebind, the same as a prepared one. Its startup output is published with the next turn, as today. Polling it early would add events after the host computed its park cut. - **Next turn** (`acpx_provider_session.rs`): `start_turn_timed` restarts only when the generation ran a turn. When `restarted_after_turn_from` is set, it does the post-restart receipt and ledger rotation without a second restart. `session.reconciled` reports the reaped process as `previousProcessId`. - **Park**: `prepare_next_generation` restarts (with the 20 s park deadline) only if no turn-end restart ran. Otherwise it marks the generation for `run.attach` rebind and returns `false`, so the runner does not ask for a second park proof. The park-time receipt rotation moves to the next turn start, where the turn-end path does it too. - **Restart failure**: the executor sets the same state that a failed turn-start restart leaves (`closed`, no session, provider exit confirmed). The next `turn.start` fails with the same `ACPX turn start failed closed: …` error that the turn-start restart gives today. - **Fake sidecar**: new `pending-input` fault, which opens an input request in a turn that then completes. ### Races | Race | Handling | | --- | --- | | A turn or steer arrives during the turn-end restart | The runner loop is single-threaded. The command waits in the socket until the restart returns, then starts on the fresh generation without a restart. Only one generation is live. | | A turn arrives before the hook runs (for example, in the same read as the ACK) | Control frames go first, so the turn restarts at its start, as today. The hook then restarts after that turn. | | Park arrives during the turn-end restart | Host control is read at the top of the loop, so park waits. Park then reuses the fresh generation. No event is published in between, so the host's park cut still matches. | | Park arrives before the hook runs | Park restarts with the 20 s deadline, as in #20145. The hook then finds a fresh generation and does nothing. | | Eviction, `runner.suspend` or retire after the restart | They reap the fresh generation through the ordinary paths. | | The restart fails | The session is `closed`. The next turn fails with the same error and the same durable state as a failed turn-start restart. Park returns conflict, as it does for a closed session today. | | Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after the restart return `already_settled`, as after any settled turn. A cancellation that retires the runtime closes the session, so no restart runs. | | Interrupted, cancelled or failed turn | `completed_turn_observed` is false, so no turn-end restart runs. The next turn restarts at its start, as today. | | Pending permission or input requests at the terminal | The reducer ends them at the terminal, so the restart runs. A later `request.resolve` fails with `no active turn`, with or without the restart. | | Active session goal at the terminal | No restart, because the goal can keep working after its prompt. | ### Removed as redundant, and what stays - **Removed:** the park-time restart and dormant verify when a turn-end restart already made the generation fresh. The runner's `host_park_ready` proves dormancy just before `prepare_for_park`, and nothing runs in between. - **Removed:** the runner's second park proof in that case, because the provider did not change. - **Removed:** the park-time receipt and ledger rotation. The next turn start does it once, for both paths. - **Kept: the turn-start restart.** Without it, a turn queued before the hook, a turn after an interrupted or cancelled turn, a turn after an active goal, and any non-Claude ACPX turn would run in the generation that ran the previous turn. Tests cover the queued-turn and interrupted cases. - **Kept: the park-time restart.** Without it, a park that arrives before the hook would leave the next run to restart at attach. A test covers this. - **Kept: the settlement of an abandoned park observation.** Without it, the restart drops events that the old generation sent before the abandoned response. A test covers this, and it fails with the settlement removed. - **Kept: the goal-dormancy check.** Without it, the restart kills an active goal's autonomous work. A test covers this, and it fails with the check removed. - **Kept: the ACK gate.** Without it, the restart can run before the host has the terminal result. A runner test covers this. - **Kept: deferred polling of the fresh generation.** Without it, its startup event lands after the host computed its park cut, and the park conflicts. - **Kept: the kept failure message.** Without it, the next turn fails with `ACPX session is not open`, and the restart error is lost. ## Verification - `cargo test -p paperclip-runner-core --lib` and every `tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery` suite: all pass (debug). - New tests (`tests/support/acpx_turn_end_restart.rs`, run from `acpx_park_preparation`) prove these points: - A settled turn reaps the old pid before it spawns the new one, and no predecessor is still live. A same-run follow-up then sends only `turn.start` and reports the reaped pid in `session.reconciled`. - Park after a turn-end restart sends no sidecar command and returns `false`. Attach rebinds the same pid. The run's second turn restarts at its end, and the third turn starts without a restart. - The reaped generation's late callback never reaches the next turn. - A turn that arrives before the hook restarts at its start. A park before the hook restarts once. - A failed turn-end restart gives the same next-turn error, retry error, durable state and process count as a failed turn-start restart. - Cancellation after the restart is `already_settled`. `runner.suspend` reaps the fresh generation. - An active goal skips the restart. An input request open at the terminal behaves the same with or without the restart. - `acpx_provider_turns`: a session-level turn-end restart drops the old generation's late `issues.delete` callback, which reuses the previous turn's call ID. The next turn then admits that call ID from the fresh provider. Interrupted and retired turns do not restart at turn end. - `acpx_park_readiness`: a turn-end-restarted generation is park-ready. An abandoned park observation tail is delivered before the restart. - `durable::runner`: the hook runs only after the terminal is ACKed and when control is idle. It never runs during ACK debt, a waiting control frame or a disconnect. - Server integration tests (`plugin-runner-release-resume`, `plugin-runner-eviction` and `plugin-runner-task-warm`) with this branch's Runner: 49 passed. The provider process counts do not change: each released run still adds one process start, now at turn end instead of at park. - `cargo fmt --check`: clean. - Fake-sidecar timing (release build, 7 runs, medians): | | before (`master` with #20145) | this PR | | --- | --- | --- | | same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms | | second turn after an attach, `turn.start` → `turn.started` | 270.3 ms | 23.5 ms | | turn-end restart, after the terminal is ACKed | — | 29.0 ms | | park preparation after a settled turn | 34.1 ms | 0.03 ms | - Real Claude: not run for this pull request. With real Claude, the restart took about 1.5 s on the turn-start path, and this change takes it off that path. That number is not measured on this branch. ## Risks - A run that ends after one turn without a park now starts one extra sidecar that no turn uses. The Claude CLI itself starts only at the first prompt, so the cost is the sidecar's start and bootstrap. - The restart blocks the Runner loop for its length (about 1.5 s with real Claude), as the turn-start restart does today. It has no deadline, as before. A park that arrives during it waits, inside the host's 30 s park wait. - The fresh generation is not polled until its next turn. If it exits while idle, the next turn finds out at `turn.start` and fails closed. Park finds out at its observation. - The turn-end restart is for Claude only, the same as park preparation. Other ACPX agents keep restarting at turn start. - If the Runner crashes during the restart, the durable state still says `session_open`. Recovery is the same as for a crash while idle. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2d93413085 |
perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path
> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.
## Linked Issues or Issue Description
No public issue exists. The issue is described inline.
### What happened?
On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.
### Expected behavior
The respawn happens while the Runner parks, so the next run starts on a
prepared generation.
### Steps to reproduce
1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.
### Paperclip version or commit
`master` at
|
||
|
|
b54317d3fb |
fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.
## Linked Issues or Issue Description
**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.
**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.
**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.
**Paperclip version or commit**
Staging revision
|
||
|
|
cedf335c98 |
fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records that turn as failed with `provider_turn_lost_on_restore` and continues on a same-conversation continuation turn > - Restart recovery blocks a run as `provider_checkpoint_permanently_failed` when its checkpoint carries a failed run terminal and no pending restart interruption > - While a continuation is running, the checkpoint keeps the lost turn's failed run terminal and names the continuation as the active turn. The pending-interruption check returns nothing once a turn is active > - So a second restart during the continuation read the run as permanently failed, blocked replay, and needed the board > - This pull request treats that failed terminal as the restart loss it records whenever an active turn follows a restart-lost terminal turn > - The benefit is that runs survive repeated restarts during their continuations, including M1 parents and their subtasks ## Linked Issues or Issue Description Refs #20148 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master |
||
|
|
b43cccbb08 |
fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the runner reports it as `provider_turn_lost_on_restore` and the controller starts one same-conversation continuation. A later restart that loses the continuation is answered the same way > - With back-to-back restarts, the controller's checkpoint can miss a continuation that the runner already accepted. On the next restore the controller starts a fresh continuation, and the restored runner then reports the unrecorded earlier continuation as lost > - The event consumer treated that stale lost terminal as the run's terminal, so the run failed while its new continuation was already running > - Recovery then refused to replay the failed run, and the task needed a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but the parent stayed `blocked` on that subtask > - This pull request makes the consumer skip a restart-lost terminal for any turn other than the continuation it started > - The benefit is that back-to-back restarts no longer turn a recoverable lost turn into a failed run that needs Retry ## Linked Issues or Issue Description Refs #20107 **What happened?** Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade between builds plus repeated restarts) at |
||
|
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
|
15b987d9b8 |
fix(runner): a finish or block waits for held calls before it is judged (#20108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Restarts must be invisible to a task: an agent that was working
keeps working after the control plane comes back
> - Restart chaos t05 (Daytona warm wave, codex, K4 hold) ended a task
as blocked even though its document save succeeded
> - #20069 refuses a finish or block that the agent decided before it
saw results that had already returned. Here the results had not returned
yet when the block was checked: after the reattach, the held calls and
the later block are handled at the same time
> - This pull request makes a finish or block wait, for a bounded time,
for earlier calls of its turn that have not returned, and then apply the
same check
> - The benefit is that a controller outage during a Paperclip tool call
no longer turns into a false "blocked" task
## Linked Issues or Issue Description
Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261108T192459-s9501`, task t05 (Daytona warm, codex, C1, K4 hold of
182 s, master
|
||
|
|
a8bbdf237c |
fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry
## Linked Issues or Issue Description
Refs #20089
**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.
**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.
**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".
**Paperclip version or commit**
master at
|
||
|
|
a88e23fa34 |
fix(runner): read an uncommitted turn start as empty history when the event window is complete (#20107)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run must survive repeated control-plane restarts; each
restore reconciles the turn its checkpoint names as active
> - That reconciliation reads the turn's history. When the provider has
no paginated history, the runnerd transport rebuilds it from the
committed runner events
> - After back-to-back hard restarts, the checkpoint can name an
accepted turn whose start was never committed. The runner was killed
after journaling `turn.start` and before the provider emitted anything
> - The rebuild treated every missing start as lost history and failed
with `codex_history_incomplete`, so the task needed a Retry
> - This pull request treats a missing start as empty history when the
committed window is provably complete
> - The benefit is that a run survives back-to-back restores, as it
already survives one
## Linked Issues or Issue Description
Refs #20074
**What happened?**
In restart-chaos wave w-20261108T200620-s9701 (local, K4/K6 + M1,
back-to-back K4 restores), three runs failed on the second restore. Two
were Claude ACPX C1 tasks; the third was an M1 parent with its subtask.
They failed with `native_session_recovery_failed:
HarnessReconciliationError: codex_history_read_failed: thread/items/list
requires supported paginated history; Error: codex_history_incomplete:
requested turn start is outside the retained runner event window`, and
each needed a Retry.
The sequence:
1. The first restore lost the in-flight turn and started a continuation
turn. ACPX echoes the requested turn id, so the checkpoint named it as
active.
2. Six seconds later a second K4 SIGKILL killed the runner. The runner
had journaled that `turn.start` (now `execution_indeterminate`), but the
provider had emitted nothing, so no `turn.started` was committed and the
runner outbox was empty.
3. On the second restore, `thread/items/list` for that turn hit the
guard, although the committed window was complete: source sequences
1–25, nothing evicted.
**Expected behavior**
The second restore treats the never-started turn like the first restore
treated its lost turn. The runner reports it lost on restore, and
recovery starts a continuation with no Retry.
**Steps to reproduce**
1. Have a checkpoint name an accepted turn whose `turn.started` was
never committed. For example, kill the runner after `turn.start` and
before the provider emits anything.
2. Restore the session. Before this change, recovery fails with
`codex_history_incomplete`.
**Paperclip version or commit**
master at
|
||
|
|
a9ae1b69c3 |
fix(runner): let a finish through after a restart lost the provider turn (#20088)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner relays an agent's Paperclip tool calls and keeps them across control-plane restarts > - #20069 refuses a `paperclip_finish` or `paperclip_block` decided before an earlier returned result reached the agent > - The guard filters by turn id, but each committed event carries the run's one identity turn id, so provider turns are not separated > - When a restart loses a provider turn, a call of that turn gets its result but never its provider completion > - The guard then refuses every later finish of the run, so the run exhausts its retries and the user must click Retry > - This pull request makes a new provider turn end the pending calls of the previous turn > - The benefit is that an agent can finish after a restart that lost its turn ## Linked Issues or Issue Description Refs #20069 **What happened?** A restart chaos storm (local, Codex, A1/C1, K2 drains for 30 minutes, 116 tasks) had one failure. Task t110 needed a user Retry. Its run was drained twice. The first drain lost the provider turn while a `read_agent_instructions` call was pending. After that, all four `paperclip_finish` calls failed with `terminal_call_before_results` and named `read_agent_instructions`. The run ended with `native_session_retry_exhausted`. **Steps to reproduce** 1. Start a Codex run that calls a Paperclip tool. 2. Drain the control plane while that call is pending, so the Runner restores the session and reports `provider_turn_lost_on_restore`. 3. Let the agent continue in the new turn and call `paperclip_finish`. **Expected behavior** The finish is recorded. The lost call belongs to a turn that the agent will not see again. **Actual behavior** The Runner refuses every finish with `terminal_call_before_results`. The run fails and needs a user Retry. ## What Changed - `unseenReturnedToolCalls` forgets the calls it tracked when a `turn.started` event begins a new provider turn. - A new test reproduces the t110 event order: a call in a lost turn, then a new turn that reads, sees its result, and finishes. ## Verification - `npx vitest run src/live/terminal-call-freshness.test.ts` in `packages/paperclip-runner`: the new test fails before the change and passes after it (7/7). - `npx vitest run src/live/runnerd-codex-transport.test.ts -t terminal`: 10 passed. - `tsc --noEmit -p packages/paperclip-runner`: clean. - Evidence: the kept Runner control-plane state of t110 shows the call `exec-0f3d1b90` (generation 1) with a `semantic_tool.result` command but no `tool.execution.completed`, followed by `turn.failed provider_turn_lost_on_restore` and new `turn.started` events. ## Risks - The t03 case of #20069 stays covered: the stale block and the earlier calls are in the same provider turn, with no `turn.started` between them. The existing t03 tests still pass. - If a driver emits `turn.started` inside one provider turn, the guard can miss a stale call of that turn. No current driver does that. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c55af26821 |
fix(runner): treat a replayed start of a turn lost on restore as history (t48) (#20074)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run must survive a control-plane restart and continue its work > - #20057 made recovery adopt an in-flight turn, or start a new turn when the restored runner reports the old turn lost > - One case was not covered. A graceful drain can detach the controller after the runner accepted `turn.start` but before the provider started the turn > - The runner then records the provider's `turn.started` while no controller is attached. After a restart it replays that start, and the Codex session rejects it as an unexpected turn > - This pull request treats that replayed start as history from the earlier generation > - The benefit is that the run starts its continuation turn instead of stopping with `native_event_replay_conflict` ## Linked Issues or Issue Description Refs #20057 **What happened?** A restart chaos wave ended with one failed task (t48, Codex). A graceful SIGTERM suspended the controller 300 ms after the runner accepted `turn.start`. The provider's `turn.started` came after the controller detached. Recovery classified the run as `resume_dead_runner`. The restored runner reported the turn as `provider_turn_lost_on_restore`, which is correct. But it also replayed the undelivered `turn.started`. The Codex session failed with "Provider started an unexpected turn." (`turn_binding_mismatch`), and recovery stopped with `native_event_replay_conflict`. **Expected behavior** Recovery starts a new continuation turn. The replayed start of the lost turn is recorded as history, and the run completes. **Steps to reproduce** 1. Start a Codex native run. 2. Detach the controller gracefully after the runner accepts `turn.start` and before the provider emits `turn/started`. 3. Let the runner record `turn.started` while it is unattached, then kill the runner and the provider. 4. Restore the run. Before this change, the session fails with `turn_binding_mismatch`. **Paperclip version or commit** master at |
||
|
|
f679cf81dc |
fix(runner): refuse a finish or block decided before earlier results reached the agent (#20069)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restarts must be invisible to a task: an agent that was working keeps working after the control plane comes back > - Restart chaos t03 (Daytona warm wave, K4 holds) ended a task as blocked even though its document was saved > - The cause is in how a terminal decision (paperclip_block or paperclip_finish) is handled after a long controller outage, not in how the Runner journals tool calls > - This pull request refuses a terminal call that the agent made before it had seen results that already returned, so the agent continues with those results > - The benefit is that a controller outage no longer turns into a false "blocked" task ## Linked Issues or Issue Description Restart chaos t03, wave `w-20261108T142906-s9501` (Daytona, codex, A1, three back-to-back K4 holds of 136 s, 252 s and 152 s). Refs #19831 (durable ownership plan). **What happened** The kept instance's run log shows the order of events: 1. The turn is accepted at 14:29:57. The runner then delivers nothing to the controller until 14:39:27. The holds keep the controller down, and the Runner keeps Codex's `item/tool/call` requests until it is reattached. 2. During that outage Codex keeps running. Its `write_document` call stays pending. The model is sampled again about once a minute and says "The save request is also waiting on Paperclip". It then calls `paperclip_block`, with the reason `tool_unavailable`. 3. After the reattach the Runner delivers the backlog in order. `write_document` runs and returns (the document is created: sourceSeq 152). Then the stale block arrives (sourceSeq 247) before Codex has acknowledged the write result: there is no `tool.execution.completed` for it before the block. The task ends blocked. **Codex behavior, checked directly** against the pinned Codex 0.160 app-server. A local probe sent one dynamic tool call and left it unanswered: - Codex does not time out. It re-samples the model at about 60 s and 127 s ("The save call is still waiting to return"). - A reply sent 130 s later is accepted (`durationMs` 130003), and the model finishes ("DONE"). So the reply is not lost. The agent's terminal decision was made without it. **Expected behavior** - A terminal call decided while earlier Paperclip results were still unseen is refused with a retryable error. - The agent continues with those results. In t03 it would see that the document was saved, and finish. ## What Changed - `packages/paperclip-runner/src/live/terminal-call-freshness.ts`: `unseenReturnedToolCalls` reads the turn's committed run events up to the terminal call. It returns each dynamic call that has a returned `tool_result` but no provider `tool.execution.completed`. `staleTerminalCallResult` builds the retryable `terminal_call_before_results` error, which names those calls. - `runnerd-codex-transport.ts` `#handleSemanticToolInput`: when such calls exist, `paperclip_finish` and `paperclip_block` get that error instead of being dispatched. The recovery path after a restart uses the same handler. A long-running call with no result yet does not count. - Crash point `runner.semantic_tool.after_input_before_dispatch`, just before the controller dispatches a dynamic tool call. A crash-point hold (`--hold-ms`, `--crash-point-hold-ms`; a K4-style receipt) keeps the controller down after the self-kill. Together they reproduce t03 on Daytona. ## Verification - Red/green: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "refuses a terminal call decided"`. - The case `seen: false` failed on master: the handler was called and the stale block was dispatched. It passes with the fix. - The case `seen: true` (the agent saw the result first) dispatches normally, both before and after. - `npx vitest run src/live/runnerd-codex-transport.test.ts`: 289 passed. `src/live/terminal-call-freshness.test.ts`: 5 passed. - `npx tsc --noEmit -p packages/paperclip-runner` passes, and `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Shared registry: 15 passed. Call-site test: 3 passed. Chaos runner tests: 12 passed. - Codex probe (local, about 2.5 min per run): reproduced the "still waiting" re-sampling and the accepted late reply described above. - Not yet run: the Daytona crash-point repro with a hold. It needs the shared harness slot, and I'll add the result here. The fix is controller TypeScript, which is not an input of the Daytona image, so the current image still matches. ## Risks - A refused terminal call costs the agent one more turn step. It applies only when a result returned that the agent had not seen, which in practice happens only after a controller outage. - The check reads committed run events. If the store evicted the earlier call's events, it finds nothing and keeps today's behavior. - Upgrade-safe run state: no persisted shape changes. The check reads events the Runner already commits. A run saved by the previous release resumes with the same history and the same answer. - ACPX and other providers emit no dynamic `tool.execution.*` events, so the check is a no-op for them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
66a2cbab98 |
fix(runner): adopt the in-flight Codex turn after a restart lands just after turn.start (#20057)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run survives a server restart by resuming its durable provider session and adopting the turn that was in flight > - The adoption rule (RC-59) only adopts a turn named by the latest `turn.start` command's accepted result > - Two gaps made it fail. A restart just after `turn.start` leaves that command `execution_indeterminate` (the runner died after journaling it). Separately, the rule demanded that Codex echo the requested turn id, which it never does. > - The restored session then rejected Codex's own in-flight turn as unexpected, and the user's Retry failed on the first click > - This pull request adopts exactly the turn the runner reconciled for that ambiguous start, and accepts Codex's own turn ids > - The benefit is that a restart landing right after a turn starts no longer costs the user a second Retry ## Linked Issues or Issue Description Found by the restart-chaos campaign: wave `w-20261108T134237-s9401`, task t77 (Codex C1), Retry run `5358f1cb`. Related: #19852 (fast follow-up turns), RC-59 adoption. No duplicate PR was found. **What happened?** The Retry run submitted its first turn at 14:05:25.171. A K2 SIGTERM followed at .690 (graceful drain, then `resume_dead_runner`). - The runner had journaled the `turn.start` and then died, so on restore it settled the command as `failed / execution_indeterminate`. - The runner's restored Codex session reconciled the ambiguous start to the live replacement turn (`session.reconciled`, `activeProviderTurnId`). - Codex then sent `turn/started` for that turn. The restored driver had no active or pending turn, because `controllerAcceptedTurnId()` was null, and failed with `turn_binding_mismatch`. - The run ended `native_event_replay_conflict`, and the user needed a second Retry. **Expected behavior** The restored session adopts the runner-reconciled in-flight turn and finishes it. It does not treat it as a replay conflict. **Steps to reproduce** 1. Start a native Codex run. 2. Send SIGTERM to the server about 0.5 s after the first `turn.start`, before runnerd confirms it. 3. Let the server restart and resume the run (`resume_dead_runner`). 4. Before this PR, the run fails with "Provider started an unexpected turn." **Paperclip version or commit** `master` before this PR. ## What Changed - `controllerAcceptedTurnStartId(commands, options)` (runnerd-codex-transport.ts): - **Codex and OpenCode results.** These providers assign their own turn ids, so an accepted result never echoes the requested `turn_lab_` id, and RC-59 adoption never matched a real Codex result. For them, the accepted turn is now adopted only while it is also the runner's live active turn in its authenticated `session.snapshot`. A turn the restored runner reports lost has no live turn, so it keeps the established restart-continuation path. ACPX still requires the echoed id. - **Indeterminate start.** When the latest `turn.start` (after the latest attach) failed with `execution_indeterminate`, it adopts the runner's live active turn. Runnerd reports one only after it reconciles an ambiguous start to a distinct replacement turn. - The transport records the snapshot's active turn at `thread/read`, which `recover()` reads before it evaluates adoption. - The driver's existing RC-59 guards still apply: the turn must be in provider history, be the only active turn, and not already be terminal. ## Verification - Connected restart test, `runnerd-restart-continuation.test.ts` "adopts the in-flight turn when a restart lands just after turn.start (t77)". It uses the real runnerd and fake Codex: - the runner and provider die after journaling a started turn; - runnerd's own restore settles the command `execution_indeterminate`; - recovery adopts `provider-turn-1` with no second `turn/start`. - It fails without the fix: nothing is adopted. - Unit cases for `controllerAcceptedTurnStartId`: - a Codex-assigned id is adopted only while it is the live turn; - an indeterminate start is adopted with the reconciled turn; - it fails closed with no live turn, an ordinary failed start, a later attachment, or an ACPX id mismatch. - Regression guarded: "starts a new turn when the restored runner reports a lost turn the checkpoint never bound" still passes. - `runnerd-codex-transport`, `codex-app-server-driver.recovery`, `runnerd-codex-transport-adoption`, `runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation` and `runnerd-controller-outage`: 383/383. Runner TypeScript typecheck is clean. - Live: the chaos campaign will run the `runner.turn_start.*` crash points (Codex A1/C1) and a Codex C1 K2 storm, and post the result here. ## Risks - **Previous-release story.** No persisted shape changes. Commands journaled by the previous release carry forward and are read with the new rule. An indeterminate start that the runner did not reconcile to an active turn still fails closed, as before. - **Wider Codex adoption.** Codex adoption now actually fires. If runnerd ever returned an accepted result that names a turn other than the one this command started, the driver would adopt it. Runnerd binds the result to the exact command id, so this is the same trust the rule already placed in ACPX. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
946c098bf3 |
fix(opencode): bind the attached run's contract after a restored launch (#20001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native OpenCode agents run through runnerd and a small OpenCode proxy. The proxy checks each `paperclip_finish` claim against the run's completion contract. > - Each follow-up run has its own contract revision. #19874 lets runnerd keep the OpenCode process across a contract change, and sends the new contract on `turn/start`. > - When runnerd starts again for a follow-up, it restores the saved session first. The restored proxy launches with the saved run's contract, but its environment already names the new run. > - The proxy applied the `turn/start` contract only when the run id changed. Here the run id did not change, so the proxy kept the saved contract and refused every correct finish claim. > - This pull request makes the proxy rebind whenever the turn's contract differs from the one its session holds. The stale-revision error also names the expected revision. > - The benefit: a queued-comment follow-up finishes on its first try, and an agent that sends a stale revision can repair it. ## Linked Issues or Issue Description Refs #19874 (warm runs carry their own completion contract). **What happened** A user sent a follow-up comment while an OpenCode agent was working. After the first run finished, Paperclip started a continuation run for the queued comment. The agent updated the plan and called `paperclip_finish` with the current revision `2`. The tool refused it twice with "Semantic result completion contract revision does not match". The error did not name the expected revision. The agent searched for it for 3 minutes, and cleanup cancelled the run. Root cause: runnerd restores a saved `session_open` session before it applies `run.attach` (`provider_backend.rs`: "execute() restores the durable provider before dispatching run.attach"). The restore launches the OpenCode proxy with the saved contract (revision `1`). Runnerd's environment already names the new run, so the proxy's `PAPERCLIP_RUN_ID` is the new run. `attach_run_in_place` keeps this provider for a changed contract (#19874) and sends `paperclipCompletionContract` revision `2` on `turn/start`. The proxy compared only the run id. The run id matched, so the proxy skipped `attachRun` and kept revision `1`. **Expected behavior** The continuation's first `paperclip_finish` with the current revision is accepted. If a claim has a stale revision, the error names the expected revision. **Steps to reproduce** 1. Use a native OpenCode agent with warm follow-ups (the default). 2. Start a task. While the agent works, post a follow-up comment, so that it queues. 3. Let the first run finish. Paperclip starts a continuation run with completion revision `2` and restores the saved OpenCode session. 4. The agent calls `paperclip_finish` with `contractRevision: "2"`. 5. Before this fix, the tool refuses it with "Semantic result completion contract revision does not match". The chat eval `steering.discard-queued-message` (runner-opencode) shows this in run events: `session.resumed` carries revision `2`, and both `paperclip_finish` results fail with the mismatch. ## What Changed - `opencode-app-server-proxy.ts`: record the contract that the session opened with. On `turn/start`, call `attachRun` when the run id changes or when the supplied `paperclipCompletionContract` differs from the session's contract. - `opencode-proxy-task-envelope.ts`: add `sameOpenCodeProxyCompletionContract`. - `opencode-server-driver.ts`: the stale-revision error now ends with `Use contractRevision "<expected>".`, like the criteria error. - Tests: a real-proxy regression for the restored launch, a driver test for the new error text, and a `repair-revision` mode in the fake OpenCode fixture. ## Verification - `pnpm exec vitest run src/cli/opencode-proxy-input.test.ts src/drivers/opencode/opencode-server-driver.test.ts src/cli/opencode-app-server-proxy.test.ts src/cli/opencode-proxy-thread-read.test.ts` in `packages/paperclip-runner`: 88 passed. - Red/green: without the source changes, both new tests fail. `binds the current contract when a restored launch already names the attached run` never sees the finish call reach the controller. `names the expected revision in a repairable stale-revision tool error` does not find the expected revision in the error. - `npx tsc --noEmit -p packages/paperclip-runner`: no errors in the changed files. (My worktree is missing a link for `@paperclipai/paperclip-eval-kernel`, which causes unrelated errors in `src/eval/workflow-harness.ts`.) - End-to-end: the chat eval `steering.discard-queued-message` (runner-opencode) with this fix cherry-picked. The run link goes in a comment below. ## Risks - Previous-release story (`rules/upgrade-safe-run-state.mdc`): this change does not change a persisted shape. The regression test uses the saved-state shape: launch parameters from the saved run, environment and `turn/start` from the new run. A runnerd that does not send `paperclipCompletionContract` behaves as before, so the proxy keeps its launch contract. The proxy and runnerd ship in the same runner artifact. - The proxy can now call `attachRun` for the same run id when the contract changes. `attachRun` resets the per-run result state. This occurs only at `turn/start`, when no turn is active, and only when the contract is different. A result for the earlier contract is not valid for the new contract. - The error text changed. No code parses it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e7e870cfc4 | fix(native): decide warm and adopted Runner process reuse from one key (#20005) | ||
|
|
65f41593d1 |
fix(runner): tell agents a skill found in a directory is imported, not re-created (#19983)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can add skills to the company library in two ways: author new content with `create_skill`, or import an existing skill so its source is kept > - #18717 told agents in the `create_skill` contract to import "an existing skill" instead of copying it > - An agent read "existing" as "already in the company library" > - It searched the library, found no match, and saved a managed copy of a SKILL.md from a vetted local directory > - The copy has no source locator, so update, audit and reset flows cannot follow the source > - This pull request says in the same contract that a SKILL.md in a local directory, repository, URL or catalog is an existing skill even when the library has no match > - The benefit is that agents import such skills and keep their provenance ## Linked Issues or Issue Description Refs #18717 **What happened** A person asked the CEO agent: "Find a useful accessibility-review skill and add it to our company." The conversation pointed to a vetted local directory with a SKILL.md. The agent called `search_skills`, found no match, and wrote in its reasoning "No existing accessibility skill. Now create it via paperclip_create_skill". It saved a `managed` skill with `sourceLocator: null`. It did not call the import API that the `create_skill` description recommends. **Expected behavior** The agent imports the skill from its directory through `POST /api/companies/{companyId}/skills/import`. The saved skill is `local_path` with the directory as its source locator. **Steps to reproduce** 1. Put a SKILL.md in a directory that a company project workspace allows. 2. Give the directory path to the agent in chat, and ask it to add the skill to the company. 3. Look at the saved skill. Before this change, the agent can save a `managed` copy without a source locator. ## What Changed - Add one sentence to the `create_skill` description in `packages/paperclip-runner/src/protocol-actions/create-skill.ts`: a SKILL.md in a local directory, repository, URL or catalog is an existing skill, even when the company library has no match, so import it instead of copying its content. - Regenerate the semantic action catalog, the semantic tool contracts, the protocol manifest digest and the seeded catalog digest. - Extend the existing contract test so that the discovered, live and scenario descriptions all contain the new rule. ## Verification - `pnpm exec vitest run src/semantic-tools/paperclip-semantic-tools.test.ts -t "distinguishes authored skills"` in `packages/paperclip-runner` passes. The new assertion fails on the parent, because the parent description does not contain the sentence. - `node scripts/generate-semantic-action-catalog.mjs --check`, `node scripts/generate-protocol-manifest.mjs --check` and `node scripts/generate-semantic-contracts.mjs --check` pass. - End-to-end chat eval `skills.find-and-install` (runner-opencode, local), with this commit cherry-picked onto the eval branch: before, run 42448994428 failed because the agent saved a managed copy. After, run 42450246200 passed. The agent quoted the new sentence in its reasoning, imported the directory through the import API, and the saved skill is `local_path` with the vetted directory as its source locator. ## Risks - Low risk. The change is model-facing text only. It does not change permissions, routes or tool inputs. - Agents that author truly new skills still use `create_skill`. The sentence applies only when a SKILL.md already exists somewhere. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
051834bdc8 |
feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - Random restart-chaos waves found these 20 ms–1 s windows only by luck > - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable write boundaries > - Nothing yet runs a task, kills the server at a chosen point, restarts it, and checks the plan's invariants > - This pull request adds that on-demand suite, as a local command and as a manual CI workflow (not a required check, per Decision 2) > - The benefit is a repeatable work list: which boundary breaks which invariant, for Phases 2–4 ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only reconciler used for I1–I7). Stacked on #19938 (call sites), which is stacked on #19937 (helper). **Problem** The named crash points cannot be exercised end to end. No command arms one point, runs real agent tasks across it, restarts, and grades the result. **Expected behavior** `node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters <list>` runs one wave per selected point and reports each point as pass, fail (with failing R rules and violated invariants), not reached, or harness error. A `workflow_dispatch` workflow runs the same thing on demand. ## What Changed - **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the shared helper and the runner mirror. Each hit reads `<token> <name>[:<nth>]`, and a new token resets the count. It is test mode only: `NODE_ENV=production` ignores it, and the disarmed path is still one comparison. - **`tests/restart-chaos/main.ts --crash-point <a>:<nth>,<b>:<nth>,...`** (config, types, ledger): - one booted server and no Vite UI; - per point: take an invariant baseline, launch the point's batch, arm the point, then on rest disarm it and check invariants as violations that are new since the baseline; - every task is graded R1–R6 and T; - `crash-point.json` (v2) holds one entry per point. - **`tests/restart-chaos/supervisor.ts`:** - every spawn gets the control file; - a SIGKILL whose receipt names the child's pid is restarted like K1: the control file is emptied first, the owned tree is killed, and a replacement starts. A failed tree cleanup fails the fault; - the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a `.paperclip/.env` above the checkout otherwise suppressed all scheduling); - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`. - **`tests/restart-chaos/invariants.ts`:** - the Phase 1 report-only pass with `record: false`: two real-clock passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7; - spool kinds are read from the wave's instance; - an unfinished pass is a harness error; - each violation carries its wake and run state as evidence. - **`tests/restart-chaos/crash-points.ts` and `scripts/crash-points.mjs`:** - select points by glob, kind and environment, and run them in one wave; - a per-point outcome: pass, fail, not_reached or harness_error; - `SUMMARY.md` with a violation evidence column. - **`.github/workflows/crash-points.yml`**, manual only, under the paid-workflow contract: - a credential-free `authorize` job; - a `suite` job in `runner-e2e-paid` whose first step reauthorizes; - a frozen install with `--ignore-scripts` and no shared cache; - only the selected adapters' keys; - the workflow is added to both security-test allowlists. - **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and "Built (Phase 5)" plus the first work list in the plan. ## Verification - Focused tests: - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/crash-points.test.ts`; - `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15 tests, including runtime arming in a real child process; - `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72 tests; - `tests/runner-e2e/workflow-security.test.ts`: 19 tests. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Loop line: question "how long is one crash-point loop?" | loop: `node scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s, rest about 2 min | result: one point alone takes about 3 min. Before this change it took about 15 min (one cold boot per point, Vite UI, host at load 77). - Full local run, 14 points, codex A1, 11.8 min in all: - every point was reached (the server killed itself at each one, and the replacement came back); - 11 pass; - `workspace.finalize.after_start_before_copyback` and `workspace.finalize.after_copyback_before_done` fail I2 and R6: a `workspace_finalize` operation stays running after the run succeeded; - `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its evidence shows the wake failed with "…retrying once", and the retry delivered the work, so this is the known refusal-regex false positive. - The 6 Daytona-only points were not run. ## Risks - Test harness and an on-demand workflow only. No product code changes. - `.github/**` needs a CODEOWNER review. The workflow uses provider keys, so it refuses non-default branches and actors that are not on the allowlist, like `runner-full-stack-e2e.yml`. - `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only crash-point waves. A chaos instance is never a worktree dev instance, so scheduling must not be suppressed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c0c26188db |
fix(runner): stamp event generations and record an earlier generation's events as history (#19960)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through the Paperclip Runner, which turns provider output (Codex, OpenCode, Claude through ACPX) into durable runner events > - In the restart chaos campaign, events from an earlier turn or runner generation failed the current turn about ten times (RC-15, 20, 20b, 20c, 33, 33b, 59 and 61) > - #19718, #19722, #19772 and #19783 each special-cased one variant, and other variants still fail the run > - The durable-ownership plan (section G, invariant 6) asks for one rule: every event carries its generation and turn, and an earlier generation's events are recorded as history and never fail the current turn > - This pull request stamps the runner generation on every event and applies one stale-event rule in the ACPX provider state, the Codex driver (which Runnerd also uses for OpenCode) and the pending `turn.start` check > - The benefit is that a late report about a settled or restored turn no longer stops a healthy turn, while events for a turn the runner never saw still fail closed ## Linked Issues or Issue Description Refs #19718, #19722, #19772, #19783 **What happened?** A runner or provider can report a turn after that turn settled: a resumed provider reports the lost turn's usage or terminal, an ACPX sidecar emits a last delta after the turn terminal, or a held failure of the previous turn sits in front of a new `turn.start`. Several of these paths fail the current turn: ACPX `AcpxEventScope` rejects any event for a non-active turn ("named a stale turn"), the Codex driver fails `turn_binding_mismatch` for a settled turn's late item between turns, and `runnerdPendingTurnStartFailure` reports any held non-recoverable failure, whatever turn it names. Runner events carried no generation. **Expected behavior** An event for a turn that is not active, and that an earlier generation accepted or this generation already settled, is recorded as history and never fails the current turn. An event for a turn the runner never saw still fails closed. **Steps to reproduce** ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event` for turn 1. Before this change, the provider state returns an error and the run fails. ## What Changed - Runner (`durable/state.rs`, `durable/runner.rs`): every event payload carries `generation`, the runner process generation that emitted it. A runner process that recovers state starts the next generation (`begin_recovered_generation`); it is saved with the next normal write, so recovery adds no write before the first turn. Loading stays pure. Old saved state loads as generation 0. Outbox events keep their original generation. - PRP event schemas v1–v3 declare the optional `generation` property; generated validators and the manifest are regenerated. - ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that names a settled turn of this run becomes a bounded `stale_generation_event` diagnostic (32 per state) and changes no state. Unknown turns still fail. - Codex driver (`codex-session-state.ts`, `codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the turn is not active, and it was restored from the checkpoint (an earlier generation) or settled in this generation. The classifier's stale-turn branch, the resume-usage branch, restored-terminal replay and the main fail point (`notificationNamesActiveTurn`) all use it. A terminal for an earlier generation's turn is history even when it disagrees with the checkpoint. A terminal that contradicts a turn this same generation settled is still a `conflicting_turn_terminal` integrity failure, and a provider that starts a settled turn again is still rejected. - Transport (`runnerd-codex-transport.ts`): a pending `turn.start` ignores a held failure that names another provider turn. - `DURABLE_TRANSPORT.md` documents the stamp and the rule. ## Verification - `cargo test -p paperclip-runner-core`: new `records_late_events_for_a_settled_turn_as_history` (red before), `stamps_every_event_with_the_runner_process_generation`; two existing ACPX tests now assert "recorded as history, no state change" instead of "rejected". Durable tests: 112 passed. `cargo fmt` clean. - `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests): late item, delta, terminal and workspace change for a settled turn during and before the next turn, an earlier generation's conflicting terminal after recovery, and an unknown turn that still fails. Red on master: 8 of 9 (the during-turn cases on master already dropped the event, with a different diagnostic; the between-turn and recovery cases failed the session). - `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's held failure no longer fails the pending start; the pending turn's own failure and a failed run terminal still do. - `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787 passed; 2 Codex executable-resolution tests timed out under host load and pass when run alone. - `tsc -p tsconfig.json` for the runner package, `check:protocol-types`, `check:replay-goldens`: clean. - Not run: chaos waves (the coordinator schedules them on this host). ## Risks - A late report about a settled turn is now dropped with a diagnostic instead of failing. A provider that keeps working on a settled turn could have its output ignored silently; a provider that starts a settled turn again, or names a turn the runner never saw, still fails closed. - `generation` is a new optional field. Consumers that do not know it ignore it, as the PRP contract requires. - The generation counter is written with the next normal state save, so there is no new write before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fda58d8cc1 |
fix(chat): post the complete final reply, not a 4,000-character run-log preview (#19971)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The agent's final message becomes the chat reply. > - The runner cuts every completed agent message to a 4,000-character run-log preview. The server then posts that preview as the reply. > - A long answer, for example a list of 501 open tasks, reaches the person cut off with "…[truncated]". > - PR #19959 fixed only the case where the accepted summary starts with the exact preview text. It missed runs where the final prose differs from the summary. > - This pull request keeps the complete final answer in the runner, and makes the server never prefer a preview over a complete source. > - The benefit is that the person receives the agent's complete reply. ## Linked Issues or Issue Description Supersedes #19959. **What happened** A chat agent wrote a final answer of about 4,985 characters (all 501 tasks). The posted chat reply was a 4,000-character preview that ended with "…[truncated]". **Expected behavior** The person receives the agent's complete final message. A preview of a message is never posted when a complete text exists. **Steps to reproduce** 1. Ask a chat agent (OpenCode or Codex) to list a large set of open work, so that its final answer is longer than 4,000 characters. 2. Let the run finish. 3. Read the chat reply. It ends with "…[truncated]" at 4,000 characters. Root cause: - `packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs`: the codex/OpenCode facade projection of `item/completed` and the ACPX `AssistantMessage` projection cut the final answer with `bounded_text(text, MAX_TEXT_CHARS)` (4,000 characters). - `durable/state.rs` `sanitize_value` caps every string again at 4,096 bytes before the event goes to the server. - `server/src/services/heartbeat-run-summary.ts` `resolveHeartbeatRunResponse` gives an explicit `final` agent message top precedence, so it posts the preview. ## What Changed - Runner (`provider_events.rs`): a completed `final_answer` agent message keeps its complete redacted text, up to a new 32,000-character `MAX_FINAL_AGENT_MESSAGE_CHARS` bound. Progress prose and in-flight items keep the 4,000-character preview. - Runner (`durable/state.rs`): `preserve_final_agent_message_text` keeps the complete redacted text of a completed final agent message when it fits in half of the transport frame. Otherwise the marked preview stays, so a large message never fails the frame check. - Server (`heartbeat-run-summary.ts`): `readCompletedAssistantMessageCandidate` marks a stored preview as `bounded`. The resolver never posts a bounded preview while a complete source exists. It posts a marked adapter final response or the accepted semantic summary instead (`bounded_final_agent_message_replaced_by_semantic_summary`). It posts the marked preview only when no complete text exists (`bounded_final_agent_message_without_complete_source`). This also fixes replies from runners that are already deployed. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-summary.test.ts`: 86 passed. Six new tests: a complete final message longer than 4,000 characters is posted in full; the old exact-prefix case posts the complete accepted answer; a preview whose prose differs from the summary posts the complete summary (also with chat precedence); a marked adapter final response wins over a preview; the marked preview is posted only when nothing complete exists; a stored preview candidate is marked `bounded`. Five of the six fail against the `master` resolver. - `cargo test -p paperclip-runner-core --lib -- final_answer complete_final`: 3 passed. Both new runner tests fail when the fix is reverted. - `cargo test -p paperclip-runner-core --lib`: 581 passed, 3 failed. The failures are `task_memory::tests::monitor_*` (OOM-kill monitor timing tests). They do not touch the changed code. - `cargo test -p paperclip-runner-core --test acpx_event_projection --test acpx_rich_events --test local_integrity_boundary_golden --test acpx_provider_events`: all passed. `cargo fmt --check` passes. - Evidence from the failed eval run: the `item.completed` final `agentMessage` text was exactly 4,000 characters (3,988 retained plus the marker). The `run.result.proposed` summary was complete at 4,985 characters. ## Risks - Larger durable events: a final answer can now be up to 32,000 characters (at most about 128 KB). The outbox keeps the preview when the text exceeds half of the frame size, so the frame check does not fail. - Redaction stays the same. The complete text goes through `redact_sensitive_text_values`, which the preview also used. - The server treats any final message that ends with "…[truncated]" as a preview. If an agent writes that text itself, the server posts the complete accepted summary instead. That summary is also the agent's own complete answer. - The runner part reaches users only after a runner release. The server part fixes replies at once when an accepted summary exists. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f755361ffe |
fix(recovery): deliver a steered message the restart lost before the model read it (#19962)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can steer a running Codex or Pi turn: a comment goes into the live turn instead of waiting for the next run > - The provider acknowledges the steer at once, but the model reads it only at its next step, after the in-flight tool call > - A control-plane restart in that window kills the provider with the message in memory. Restart recovery starts a continuation turn that does not carry the message. The message is lost and no error shows > - The restart chaos harness showed this in 6 of 6 steered runs (Codex, K1, K2 and K6, 2 samples each): steer acknowledged, run succeeded, no reply to the steered message > - This pull request makes restart recovery deliver every acknowledged steer that the model did not read, exactly once > - The benefit is that manual Steer no longer loses messages on restart, and the planned automatic steering can ship on top of it ## Linked Issues or Issue Description **What happened?** A comment steered into a running Codex turn got no reply when a control-plane restart (SIGKILL, SIGTERM or crash loop) happened during the agent's tool call. The run succeeded, and the steered comment had no effect. **Steps to reproduce** 1. Start a Codex task that runs a 40-second command. 2. During the command, post a comment and click Steer. The provider acknowledges it. 3. Restart the control plane before the command ends. 4. Recovery resumes the turn with a continuation. The agent finishes the task and ignores the comment. **Expected behavior** The agent reads and handles the steered comment once, after recovery. **Root cause** The provider keeps acknowledged steering input in memory until its next model step. The restart continuation (`paperclip.native-continuation.v1`) only told the model to continue the current request, so the acknowledged input was gone. ## What Changed - `unreadSteeredMessageIds` (runner package): derives acknowledged-but-unread steers from run-log events in order. A `steering_acknowledgement` item adds the message id (once per id). A `userMessage` item in the same turn reads the oldest unread steer. Codex records each steered input as one `userMessage` item when it enters the conversation. A `userMessage` item in a later turn reads all earlier steers, because the continuation carries them. `turn.completed` reads all steers. - `readUnreadSteeredMessages` (server): reads those facts from `heartbeat_run_events` (only transport-validated events with a `sourceEventId`). It also reads the comment or interaction text for the same company and issue. The run log is the durable record of both "acknowledged" and "read", so no new column is needed. - The restart continuation now includes `steeredMessages` and one instruction to handle them, oldest first. The runtime reads them through a lazy `getUnreadSteeredMessages` option, only when a restart continuation starts. Native recovery wires it for every run. - `queuedInteractionSteeringBody` is extracted, so recovery rebuilds a steered interaction response with the same text the steer sent. - Fake Codex gets `--consume-steer`, which emits the `userMessage` item for an accepted steer, like real Codex. ## Verification - Red/green live test with real runnerd and fake Codex (`runnerd-restart-continuation.test.ts`, "delivers a steered message the restart lost exactly once"): start a held turn, steer it, SIGKILL runner and provider, restore and recover. - "unread": the continuation carries the message exactly once. Without the runtime change this test fails. - "read" (`--consume-steer`): the continuation does not carry it again. - Both: after the continuation completes, a later restart has nothing to deliver. - `steering-delivery.test.ts`: ordering, dedupe, initial-input, later-turn and completed-turn rules. - `unread-steered-messages.test.ts` (embedded Postgres): ignores events without a source id and messages from other issues. It reads interaction text, ignores model-output items as read evidence, and returns nothing after the continuation input. - `pnpm vitest run src/steering-delivery.test.ts src/live/runnerd-restart-continuation.test.ts` (runner) and `pnpm vitest run src/services/native-runtime/unread-steered-messages.test.ts` (server): pass on the rebased revision. - `issue-queued-comments-routes.test.ts` and `native-session-runtime.test.ts -t restart`: pass. `native-session-resume.test.ts` "archives a damaged prior epoch…" fails locally on plain `origin/master` too (local runnerd binary); CI runs it with a fresh build. - Restart chaos (Codex steer mode, K1/K2/K6) runs after this lands in the harness slot. Before: 0 of 6 steered messages answered. ## Risks - Duplicate delivery: a provider that records steered input without a `userMessage` item, or a `userMessage` event lost in the crash window before persistence, makes recovery send the message again. The model then sees it twice. That is safer than losing it, and the instruction asks the model to handle each one once. - Pi (ACPX) steering has no `userMessage` read evidence today, so a Pi steer that the model already read before a restart may arrive again in the continuation. - Only the restart continuation carries the messages. Other recovery paths start after a completed turn, which already read every steer. - No schema or migration change. A session saved by an earlier release resumes normally; its run log already holds the acknowledgement events. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
18644a3263 |
feat(recovery): instrument 20 durable write boundaries with crash points (#19938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - #19937 added `crashPoint(name)` and a registry of 20 named boundaries from the Phase 0 inventory > - A name without a call site cannot fire, so the crash-point suite cannot test it > - This pull request puts one call at each registered boundary and adds a test that keeps the registry and the call sites in step > - The benefit is that the on-demand suite (next pull request) can kill the control plane at every known write-ahead gap ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead coverage"). Stacked on #19937 (helper and registry). Refs #19861 (ownable kind ids). **Problem** The registered crash points have no call sites. **Expected behavior** Each registered point is called at its gap: after the first durable write or outside effect returns, before the next one starts. Disarmed (always in production) each call is one comparison. **Scope** Call sites and a coverage test only. The suite runner and CI workflow follow. ## What Changed - Server call sites (`@paperclipai/shared/crash-points`): - `sandbox-creation-operations.ts`: around the provider `create` call (intent → provider → lease). - `native-session-executor.ts`: after `native.process_start_requested`, before spawn. - `paperclip-control-plane-port.ts`: after the result-accept transaction, before finalization starts (only when a result was inserted). - `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before the run projection. - `workspace-operations.ts`: `workspace_finalize` only — after the running row, and after `settle()` before the done row. - `environment-runtime.ts`: provider release → lease row; teardown claim → destroy; reusable destroy → lease release; stop-and-retain claim → stop. - `heartbeat.ts`: lease release → runtime services; budget reservation hold → dispatch. - `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`. - `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit → spool unlink; receipt spool → DB row → unlink. - `packages/paperclip-runner/src/crash-point.ts`: a small copy of the helper for controller code in this package, which does not depend on `@paperclipai/shared` (adding that dependency would change the runner content ID). It arms only `runner.*` names. runnerd never sees the variable: its environment is an allowlist. - Runner call sites: after the runnerd spawn, before its identity is published (both spawn paths); `turn.start` result received → controller journal saved → next send. - `server/src/__tests__/crash-point-call-sites.test.ts`: every registered point has a call site, every call names a registered point, and every `kinds` entry is an ownable kind id of the Phase 1 registry. ## Verification - `cd server && npx vitest run src/__tests__/crash-point-call-sites.test.ts` → 3 passed. - `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean. - Behavior with nothing armed is unchanged: each call returns at its first comparison (measured in #19937: about 3.7 ns). ## Risks - Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT` names it and `NODE_ENV` is not `production`. - Points on the path before `turn.start` (process start, runner spawn) cost nanoseconds, within plan invariant 8. - `usage-receipts.ts`: the persist-then-unlink one-liner became a block with the same statements in the same order. - The runner package change gives the runner a new content ID, as any runner source change does. No persisted run state changes (rules/upgrade-safe-run-state.mdc). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits) and a read-only exploration subagent that mapped the boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0189fedffc |
fix(native): bound an idle warm local runner whose controller died (#19879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between runs so a follow-up turn starts fast. The follow-up default flip (next PR) makes warm the default for local Codex and OpenCode agents. > - The controller owns a warm runner's idle window. Runnerd ignores the idle timeout itself. > - A graceful controller shutdown suspends idle warm sessions. A controller that dies (SIGKILL, crash) does not. Local runners launch with no process reconnect grace, so the idle runner kept running until its 1-hour connection lease ran out, owned by nobody. > - This pull request gives warm local runners the 7-minute reconnect grace that remote runners already use. > - The benefit: idle runners are reaped by proof within a bounded time, never left orphaned, before warm becomes the default. ## Linked Issues or Issue Description Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852. **What existing behavior does this improve?** Ownership of an idle warm local runner after its controller dies. **Current behavior** `runnerProcessReconnectGraceMs` is `undefined` for local targets, so runnerd launches without `--reconnect-grace-ms`. An idle warm runner whose controller was killed stays alive until its connection lease (one hour) expires. The restart-chaos harness cannot show this: its K1 kills the controller's whole process tree, runner included. **Proposed behavior** `localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7 minutes) for a warm policy and `undefined` for per-turn. After the grace, runnerd records `transport_reconnect_grace_exceeded`, stops its provider, keeps durable state and exits. The next run resumes cold. **Reason and benefit** The restart plan requires idle runners to be reaped by proof, never orphaned. Seven minutes is far longer than a controller restart (13–20 s locally) and the long-outage fault holds (3–5 minutes), so a live turn still reattaches. ## What Changed - `native-runner-recovery-contract.ts`: `localRunnerProcessReconnectGraceMs`. - `native-session-executor.ts`: the local runnerd transport passes it as `runnerProcessReconnectGraceMs`. Remote targets keep `null` (lease-owned). - `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex. An idle warm runner, after its turn completed, whose controller detaches. - `doc/DEVELOPING.md`: the grace and its bound. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the idle runner exits with `transport_reconnect_grace_exceeded` and the next transport resumes the same provider session cold. With no grace (`null`, today's local launch), the runner is still alive after the controller left. - `cd server && npx vitest run src/services/native-runtime/native-runner-recovery-contract.test.ts`: warm gets the remote grace; per-turn keeps none; the grace outlasts a controller restart and a long outage. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - Local restart measurement with warm runners (K1/K2, codex, claude, opencode, n = 3 each): every interrupted run recovered, with recovery latency equal or lower than per-turn. The chaos waves with this change are in the default-flip PR. ## Risks - A warm local runner whose controller stays down for more than 7 minutes exits. A turn in progress then resumes through `resume_dead_runner`, the same path as a controller crash that kills the runner. Remote runners already behave this way. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted state changes. A warm runner launched by the previous release keeps the launch arguments it started with (no grace) until it exits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
410aa6d1b4 |
test(runner): resolve provider fixture signals through PATH (#14476)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its native runner tests verify exact provider shutdown and recovery. > - Four fixture call sites invoke `/bin/kill` directly. > - NixOS supplies `kill` through PATH, so those tests fail before checking process state. > - This pull request resolves `kill` through PATH, as the neighboring runner tests already do. > - The same stop, probe, and resume assertions can run on non-FHS hosts. ## Linked Issues or Issue Description **What happened?** Four Codex provider tests fail with `No such file or directory` on NixOS. Each failure comes from a test-owned `/bin/kill` invocation. The executable is installed on PATH. **Expected behavior** Tests should probe and signal their disposable provider processes with the installed `kill` command. **Steps to reproduce** On a NixOS host with `kill` on PATH and no `/bin/kill`, run the `codex_provider` Rust integration target. Three `durable_stop_*` cases and `structured_question_round_trips_through_the_normalized_backend` fail. All four failures were reproduced separately on unmodified `e9debd3eac328e6abe87ab89e9ff40ee2bb56b73`. Related: these failures also blocked the broader local qualification of #14474. This fixture fix has no dependency on that runtime change. ## What Changed - Replace four test-only `/bin/kill` executable paths with `kill`. - Retain every signal, PID/process-group operand, assertion, and unwind cleanup. - Match existing PATH lookup in this file and `tests/process_supervisor.rs`. ## Verification Rebased onto master `820c458465` (single commit `ad1fb1043`). - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --test codex_provider --locked -j4 -- --test-threads=2`: 94 passed, 0 failed, 2 ignored (subprocess helper entries that their parent tests run). The four tests that used `/bin/kill` pass on a host without `/bin/kill`. - No production code or documentation changes. - Hosted CI: `General tests (workspaces-b)` timed out in `packages/db/src/backup-lib.test.ts` and `e2e shard (1/8)` hit `ERR_STREAM_UNABLE_TO_PIPE`. Neither area is touched by this test-only diff. Master pushes currently run the "CI profile skipped" workflow, so there is no master run of those jobs to compare against. ## Risks Low risk. Tests require a working `kill` on their test PATH. Fixture PIDs and arguments stay unchanged. ## Model Used - Anthropic Claude Sonnet 5.5 (`claude-sonnet-5-5`) through Claude Code with tool use and code execution, for the rebase and verification. Original change: OpenAI `gpt-6-astra` through OpenCode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
dcaada5f59 |
fix(native): carry each warm run's own completion contract (#19874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd and the provider
process alive between runs, so a follow-up message can start fast.
> - Each run has its own completion contract. The revision changes with
each follow-up, and the agent must name that revision when it finishes.
> - On a warm attachment the transport rebuilt the `run.attach` payload
from its constructor options. Those options belong to the first run.
Every later run kept the first run's contract.
> - The OpenCode proxy checks the finish claim locally against that
stale contract. Every warm OpenCode follow-up was refused ("Semantic
result completion contract revision does not match"), and the agent
looped until the run timed out.
> - This pull request carries each run's own contract into the warm
attachment, and lets runnerd adopt it without a cold provider resume.
> - The benefit: warm follow-ups finish correctly, and they keep the
live provider process, which is the reason to be warm.
## Linked Issues or Issue Description
Refs #19852 (startup sub-spans), #19846 (TTFT baseline).
**What existing behavior does this improve?**
Warm cross-run attachment for native Codex and OpenCode agents.
**Current behavior**
In a local measurement wave with `lifecycleMode: warm`, 4 of 4 OpenCode
follow-ups looped for 8+ minutes. Each finish call was refused with
"Semantic result completion contract revision does not match". The saved
`runAttachTemplate` after the attachment still had revision `1`, while
the follow-up turn named revision `2`. Codex hid the defect because its
finish claim goes through the controller's tool path, which uses the
current input.
**Proposed behavior**
The attachment carries the attaching run's contract. Runnerd updates the
contract in place for Codex and OpenCode and keeps the provider process.
The OpenCode proxy receives the current contract with the run identity
on `turn/start`.
**Reason and benefit**
Warm follow-ups are the largest TTFT lever in the baseline. A
correctness-only fix (cold-resume the provider when the contract
changes) would make every warm follow-up relaunch Codex or OpenCode,
because the revision changes on every follow-up.
## What Changed
- `attachRun` carries `completionContract` through `NativeSession`, the
harness backend, the Codex harness session and the runnerd transport.
`executeNativeSession` passes the current input's contract. The
transport uses it instead of its first-run `resumeCompletionContract`.
- Runnerd (`codex_provider.rs`, `attach_run_in_place`): a changed
contract no longer forces a cold resume for `codex` and `opencode`.
Codex app-server never receives the contract, and the OpenCode proxy now
receives it per turn. Other providers keep the cold-resume boundary. A
changed tool set still forces it.
- Runnerd sends `paperclipCompletionContract` with each OpenCode
`turn/start`, next to the existing `paperclipRunId`.
- The OpenCode proxy passes that contract to `attachRun` when the run
changes. `OpenCodeHarnessSession.attachRun` replaces its envelope's
contract and validates it.
## Verification
- `src/live/runnerd-codex-transport-warm-attachment.test.ts` (new case):
real runnerd and fake Codex. After a warm attachment with a new
contract, the saved `runAttachTemplate` has the new revision and the
Codex pid is unchanged.
- `src/cli/opencode-proxy-input.test.ts`: the real proxy bundle serves a
warm second run with revision `proxy-v2`. The second
`paperclip/runResult` is produced and names `proxy-v2`. Without the fix
the finish call is refused.
- Each test fails without its part of the fix: the proxy test without
the proxy change; the transport test without the transport change (stale
saved revision); and, with `PAPERCLIP_ATTACH_TRANSITION_RUNNER` pointed
at a runnerd built without the `codex_provider.rs` change, the Codex pid
changes.
- `cargo test -p paperclip-runner-core --lib warm` (11 tests),
`native-session-runtime.test.ts` (232), the OpenCode driver, backend and
warm-attachment suites, and `tsc --noEmit` for the runner package pass.
## Risks
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Saved `runAttachTemplate` values carry forward,
and the next attachment writes the current contract. A new runnerd with
an old OpenCode proxy behaves as today: the proxy ignores the new
parameter and keeps the stale contract. An old runnerd sends no
parameter, so a new proxy keeps its launch contract.
- In-place adoption widens the warm-retention path for Codex and
OpenCode to changed contracts. The idle, accounting and pending-work
checks still apply first.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a9ac2e2039 |
feat(native): split local runner startup into stage, launch and session-load spans (#19852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a person messages an agent, the agent must start its turn fast. Time to first token (TTFT) and agent boot are first-class metrics. > - The Phase 0 baseline (#19846) found that local runner startup costs 1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT. > - The run log records that startup as one span (`runner.session.bootstrap` or `runner.session.resume`). We cannot see if the time goes to staging, to the `paperclip-runnerd` process launch, or to the provider session load. > - This pull request splits local runner startup into three sub-spans in the run log. > - The benefit: the follow-up work to cut follow-up-turn TTFT can measure each step before and after. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside local runner startup"). Refs #19838. **What existing behavior does this improve?** The run-log performance spans for local native runner startup. **Current behavior** The local path records one `runner.session.bootstrap` or `runner.session.resume` span that covers runtime staging, the runner process launch and the provider handshake together. The remote path already records `runner.runtime.stage` and `runner.process.launch`; the local path does not. **Proposed behavior** The local path records three sequential child spans of `runner.session.startup`: `runner.runtime.stage`, `runner.process.launch` and `provider.session.load`. **Reason and benefit** The TTFT work needs to know which step dominates before it changes the runner lifecycle. Without these spans, a change to warm runner reuse cannot show which cost it removes. ## What Changed - `RunnerdCodexTransport` has a new optional `onStartupStep` callback. `#start` and `#resume` time three milestones and call it synchronously. A failing callback never changes startup. A nested startup (exact adoption from `thread/start`) keeps the outer clock. - `runner.runtime.stage`: launch preparation before the runner process exists. - `runner.process.launch`: spawn until runnerd accepts the run (`run.prepare`, or the recovery attachment on resume). An adopted live runner records no launch. This keeps the existing meaning in `summarizeStartupReuse`: a launch span means a fresh runner. - `provider.session.load`: `session.open` (or the recovery probe) until the provider identity is confirmed. - `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each step as a `run.performance.span` row with `mode` and `target: "local"`. It uses `trace.defer`, so the run-log write never delays `turn.start`. - The native executor passes the callback only for a local target. Remote launchers keep their own spans, so there are no duplicates. - `redaction.ts`: allowlist `runner.process.launch` and `provider.session.load` as native span names. - `doc/run-log-events.md`: new section "Local runner startup sub-steps". Data path (AGENTS.md §5.7): run log only, plus Observability spans when an operator sets an OTLP endpoint. The attributes (`mode`, `target`, `outcome`) are already in the closed span-attribute allowlist. No Telemetry change. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd and the fake Codex fixture. Bootstrap and a cold resume each report stage, launch and session load in order, with `ok` outcomes and no overlap. A throwing sink does not change startup. - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/__tests__/redaction.test.ts`: the rows have the correct parent, offsets, durations and failed level. A stalled run-log sink does not block the caller. The rows settle before `task.provider_session.measured`. The names survive redaction. - The existing `runnerd-codex-transport*` suites pass (412 tests). Under host load (load average 16–27), the first full run had 7 timing failures in suites that do not set the callback. Each failing file passed when I ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for the failing case alone. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Low risk. Without `onStartupStep`, the transport behaves as before: the clock is not created and each milestone call is an optional-chain no-op. - More run-log rows: three per local runner startup. - The step boundaries are controller-side observations. `runner.process.launch` includes the runnerd authentication and the `run.prepare` acknowledgement, not only the OS spawn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5b972ceb1 |
fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The local runner (`paperclip-runnerd`) supervises each provider process, such as the ACPX sidecar for Claude, in its own process group > - Run teardown stops that process group, and nothing else > - Claude Code starts each Bash tool command in a new session (`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job in that session, and the wrapper exits at once. The job is then a child of init and outside the owned group > - In a restart chaos storm, a run failed after a restart resume. Its tool job kept running beside the task's next run, in the same workspace, for more than four minutes > - This pull request makes the supervisor record every process that its provider starts, including processes that leave the group, and stop them all before teardown reports success > - The benefit is that a run's tools cannot outlive the run and cause duplicate side effects in the next run ## Linked Issues or Issue Description **What happened?** A restart chaos storm ran a local ACPX Claude task. After a restart resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo "started pid $!"` and then ended its turn without a result. The run failed with `native_finalization_missing`. The runner tore down the sidecar group, and the guardian killed the provider group. `bench.sh` was in the session that Claude Code made for the tool (pgid and sid = the exited wrapper, ppid = 1). It was never signalled. It kept forking `sleep 1` while the task's next run started and finished in the same workspace. **Steps to reproduce** 1. Run a local ACPX Claude task whose agent runs a Bash tool command such as `nohup ./bench.sh x > log 2>&1 & echo $!`. 2. End the run (any outcome, including failure after a restart resume). 3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still running with ppid 1. **Agent adapter(s) involved** Claude Code (local ACPX). The gap is in the shared supervisor, so any provider whose tools leave the group is affected. **Root cause** `SupervisedProcess` (`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`) spawns with `process_group(0)` and stops only that group in `terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its own comment said that a caller that must exclude escaped or re-parented descendants needs a separate lifetime fence. No caller had one. The ACPX sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`) and the ACPX provider guardian (`installation-integrity.ts`, which kills only its own group) therefore left every `setsid` tool job behind. The failure path after the resume was not special: the same gap applies to every teardown. **Expected behavior** When a run ends for any reason, every process that its provider or tools started stops, and teardown proves that they are gone. Identity comes from PID, group, session and start time, never from a command name. ## What Changed - Add the `paperclip-process-table` crate. It reads PID, parent, process group and start time for every process (libproc on macOS, `/proc` on Linux). `runner-core` forbids unsafe code, so the two checked libproc calls live in this small crate. - Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50 ms) records each descendant of the supervised process by PID and start time. It also records each group and session that a descendant leads, with the leader's start time. A process belongs to the lineage when its parent belongs, or when it is in a recorded group or session and started no earlier than its leader. Groups and sessions that the supervised process only inherited (runnerd's session) are never recorded. A recorded ID whose PID now names a different process is dropped. - `SupervisedProcess` samples the lineage just before it signals, and after the leader is reaped it stops every live member: TERM, then KILL after the shutdown grace. It fails with an error until it proves that no member is alive (grace plus 5 s). This applies to `terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. - If the lineage sampler cannot start, the supervisor stops the process it just started and reports the failure, so a provider never runs without tool cleanup. - Table reads and state updates happen under one lock, so an older read cannot erase members that a newer read recorded. - The process table reports processes that exist but cannot be read. A recorded member that cannot be read stays unresolved, and teardown fails until its exit is proven. - The Runner workspace now has two members. `scripts/runner-release-crate-cache.mjs` and `scripts/runner-debug-crate-cache.mjs` cache and mark current the process-table lib unit with the Runner crate (otherwise Cargo rebuilds the Runner crate after every restore). The debug contract accepts only this library-only member and still refuses any other workspace change. The workflow cache path lists and the debug clean step are updated to match. - `terminate_group_immediately` now accepts `EPERM` from the group signal when the leader has already exited. macOS returns `EPERM` for a group whose only member is the unreaped leader. The lineage pass then proves that no other member remains. ## Verification - `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job immediate_group_fence` (macOS, local). - New real-process tests in `process_supervisor.rs`: a supervised shell starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs `nohup <loop> &`, writes the job PID and exits. The test asserts that the job is outside the owned group and re-parented, then calls `terminate_group`, `terminate_group_immediately` or drops the process, and asserts that the job is gone when the call returns. - Red: with the tracker disabled, all three fail with `tool job <pid> outlived its provider`. - Green: all three pass. - Unit tests in `process_lineage.rs` cover a re-parented job, a job that left its group but stayed in the recorded session, a session made after the first sighting, PID reuse, and no claim through the supervisor's inherited session. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the process-table library is cached with the crate, and any other member still needs review). - `cargo test -p paperclip-process-table` (includes an exited process reported as gone), and `cargo test -p paperclip-runner-core --lib` (570 passed). `acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` fails sometimes under full-suite load. It fails at the same rate with the tracker disabled, so it is a flake that already existed. - Sampler cost: one table read takes about 0.4 ms for 1,200 processes on macOS (about 0.8% of one core for each supervised process). - The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did not run. The local session was not permitted to load provider credentials. It is not proven against a live Claude run. ## Risks - Teardown now stops all the processes that a provider started, also intentional background jobs. This is the intended contract: nothing outlives the run. - A wrapper that lives less than one sample (50 ms) and orphans a job before the sampler sees it is not recorded. Claude Code's wrapper sources a shell snapshot and lives for seconds. On Linux, a child-subreaper would close this gap. That is not in this change. - When `runnerd` itself is killed with SIGKILL, this code does not run. The provider guardian still kills its own group, but escaped tool sessions are not stopped in that case. - Teardown can return a new error when it cannot prove the lineage stopped. Callers already handle `terminate_group` errors as provider shutdown failures. - The server does not yet gate the next run on the tool lineage of the prior run directly. It relies on the runner finishing its teardown before the runner exits. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b2af45f0f1 |
fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.
## Linked Issues or Issue Description
Refs #19516
**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.
**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.
**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.
**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.
I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.
## What Changed
- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
- The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
- A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
- Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.
## Verification
**Red on `master` (
|
||
|
|
5eac099fbb |
fix(agents): batch independent follow-up questions into one card (#19634)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents ask users for input with the `request_human_input` tool or
the `ask_user_questions` interaction. Paperclip shows each call as one
question card.
> - In an onboarding walk, a Codex execution agent asked five
independent questions (name, menu and prices, contact, style, output).
It posted five cards, one after the other. Each card came 20-60 s after
the previous answer.
> - The tool description said: "For one question at a time, ask only the
next unanswered question and wait for its real answer." Agents read this
as a rule. The paperclip skill showed only a one-question example and
did not tell agents to batch.
> - No server cap exists. A question set accepts up to 64 questions.
> - This pull request changes the guidance. Agents now ask independent
inputs together in one card. They use a follow-up card only when a
question depends on an earlier answer. One-at-a-time stays available
when the user asks for it.
> - The benefit is fewer waits for the user: one card and one agent turn
instead of five.
## Linked Issues or Issue Description
**What happened?**
An execution agent asked a user five independent follow-up questions in
five separate question cards. Each card appeared only after the user
answered the previous card and the agent finished another turn (20-60 s
each).
**Expected behavior**
When an agent needs several independent inputs, it asks them together in
one multi-question card. The onboarding interview already does this ("1
of 4" pager).
**Steps to reproduce**
1. Complete onboarding on the Codex API path and hand off a task such as
"Create printable cake menu and logo".
2. Watch the execution agent's first turns.
3. See five sequential single-question cards.
**Paperclip version or commit**
master at
|
||
|
|
2468b3f907 |
fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart loses a turn, the native runtime sends a same-conversation continuation (#19501, #19699, #19771) > - A restart can also kill shell or background processes that earlier turns started > - In three restart storms, the continuation agent could not inspect its killed `bench.sh` and set the task to blocked > - The continuation prompt told the agent not to "blindly rerun an action whose outcome is unknown", so the agent did not re-run the command > - This pull request tells the agent that earlier processes may be gone, and to check the workspace outputs and re-run a command whose expected output is missing > - The benefit is that a task can finish after a restart instead of blocking on a process that no longer exists ## Linked Issues or Issue Description Refs #19501, #19699, #19771 **What happened?** Restart chaos workload B5 (Codex and Claude) failed this way three times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started `bench.sh` in the background. A restart killed it. The restart continuation agent tried to find its tool session and got "process inspection is denied". It then blocked the task. The comment read: "Blocked by the Paperclip runtime restart: the benchmark session is unavailable…". It added that it "cannot verify whether the original run is still active or recover its output, so I have not rerun it". **Expected behavior** The agent knows it is in a resumed session after a restart. It checks the workspace for the command's outputs. If the expected output is missing, it re-runs the command and finishes the task. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts`, `restartContinuation`: the prompt did not say that processes from earlier turns may be gone. It told the agent to inspect each unfinished command's "current state" and not to "blindly rerun an action whose outcome is unknown". With the process gone, the agent could not inspect it, so it blocked. ## What Changed - The restart continuation prompt now says: - This is a resumed session after a Paperclip control-plane restart. - Shell commands and background processes from earlier turns may no longer be running, and their tool sessions may be gone. Do not block only because one cannot be found or inspected. - Check the workspace for the outputs or artifacts the command should have produced. If the expected output is present, use it. If the expected output is missing, re-run the command. - Do not repeat completed work. Before repeating an action with external effects (a push, a message, a deployment), check whether it already took effect. If you cannot tell, report the blocker. - The prompt does not say that re-running is always safe. Process teardown (#19727) has not merged. - `doc/execution-semantics.md`: one additive sentence. ## Verification - `native-session-runtime.test.ts`: the three restart continuation tests (Codex, ACPX, OpenCode) now check the new guidance. They were red before the prompt change and are green after it. - Guard: "uses the admitted session cwd in the task envelope" now checks that fresh turns and ordinary resumed turns do not contain the restart guidance. - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/ src/live/runnerd-restart-continuation.test.ts` in `packages/paperclip-runner`: 603 passed. - `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for `packages/paperclip-runner` are clean. - Not run: a live restart-chaos replay with a real model. ## Risks - Low. This changes only the prompt text of the restart continuation. Ordinary turns do not change. - An agent may now re-run a command whose earlier run had partial side effects. The prompt asks it to check external effects first, and to re-run only when the expected output is missing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |