mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
ac02de7732e8aad2d44304792db8cec4a5a0ebc0
567
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
28c11084ad |
test(runner): wait for the background task notification turn before the next Claude turn (#22522)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner converts raw Claude ACP session updates the same way ACPX does. One test proves this against the real Claude ACP bridge. > - The test runs four turns. The third turn starts a background command (`sleep 1`). When that command ends, Claude starts its own task-notification turn. > - The test starts to record raw updates before ACPX sends the next prompt. If the notification turn arrives in that window, the test records it. ACPX correctly does not give it to the prompt. The two lists then differ. > - This race stopped the Release for |
||
|
|
611dc68581 |
fix(runner): continue a lost turn on a resumed warm session (#22447)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a managed sandbox run under the Paperclip runner (`runnerd`). When `runnerd` exits mid-run, the transport relaunches it, and the runner restores the same conversation > - #22235 lets the execution that started a turn send one continuation turn when the restored runner reports that turn as `provider_turn_lost_on_restore` > - That rule applies only to a session that the execution opened fresh (`!recovered`). A run that resumes the warm conversation of its previous run is a recovered session. A chat follow-up does this on every new message > - So a resumed session that loses its own task turn gets no continuation. The run fails, and the user must press Retry > - This pull request makes the task turn that this execution started its own turn, whether the session was opened fresh or resumed > - The benefit is that a chat follow-up keeps its conversation and workspace after a mid-run `runnerd` restore, as #22235 intended ## Linked Issues or Issue Description Refs #22235 (continue a turn lost to a mid-run runnerd restore). Refs #22068 (disk reserve). Refs #22239 (stop retrying a run that filled its sandbox disk). **What happened?** On staging, an agent in a chat task filled its 10 GiB managed sandbox disk. It fetched the full history of a large Git repository and searched it. `runnerd` exited with `failed to create private durable state temporary file: No space left on device (os error 28)`. The disk reserve let the transport relaunch `runnerd`, and the runner restored the same conversation. Then the runner reported `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}` for the turn that this run had started. The runtime did not send the continuation turn. The run failed as `sandbox_disk_full` with the message "The runner restored the conversation after process loss, but the previous turn is no longer active." A manual Retry on a fresh provider session finished the work. The run started from the warm checkpoint of the previous run in the same conversation (`startup reuse: provider_session (reused)`, then `session.resumed` with the same provider session id). `loadWarmNativeCheckpoint` rebinds that checkpoint to the new run and clears its turns. `executeNativeSession` recovers it, so `recovered` is true. The run then starts its own task turn. `restoredInThisExecution` required `!recovered`, so that turn could not be continued. **Expected behavior** When a mid-run `runnerd` restore loses the task turn that this execution started, the execution sends one continuation turn in the restored conversation. This applies to a fresh session and to a resumed warm session. **Steps to reproduce** 1. Run a native agent through `paperclip_runner` in a managed sandbox, as a follow-up on an existing conversation (a warm resumed session). 2. During the turn, make `runnerd` exit, for example by filling the sandbox disk. 3. The transport relaunches `runnerd`, and the runner reports the turn as lost on restore. 4. The run fails instead of continuing the turn. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: `restoredInThisExecution` no longer requires a fresh session. The turn id that this execution records when it starts its task turn is now enough. - The execution records that turn id only for a turn that carries the task work. A disposition-only recovery turn (the turn that asks only for `paperclip_finish` after a completed turn) is still not continued, as before. - All other guards stay: one continuation per run, the provider allowlist, no compaction or goal runs, no pending runtime requests, no saved semantic result, and the snapshot must show that same turn as restart-interrupted. The control-plane restart claim path (`restoredAtStartup`) does not change. - `native-session-runtime.test.ts`: three new cases for a recovered warm session: - it continues its own task turn once after a mid-run restore; - it does not continue a lost turn that it never started; - it does not continue a lost disposition-only recovery turn. ## Verification - Red before the fix: `npx vitest run src/native-session-runtime.test.ts` in `packages/paperclip-runner` failed `continues its own turn on a resumed warm session once when runnerd is restored mid-execution` with `NativeProviderTerminalFailure` (`code: native_provider_terminal_failed`, `providerCode: provider_turn_lost_on_restore`). This is the same failure as on staging. - Green after the fix: the same file has 303 passed, 1 failed. The one failure is `settles a governed wait through real Runner shutdown after a missing Codex terminal`. It needs the Rust `paperclip-runnerd` binary (`ENOENT runner/target/debug/paperclip-runnerd`). It fails the same way before and after the change. - The existing `requires the control plane restart claim` and `does not continue a lost turn this execution never started` cases still pass. - To prove that the disposition-only guard is necessary, I removed it locally. The new disposition-only case then failed (the run continued). I put the guard back. - `native-session-runtime.provider-turn.test.ts`: 36/36 passed. The Codex driver provider-turn, recovery, stale-generation and terminal-replay suites and `terminal-call-freshness.test.ts` passed. - Not run locally: `src/live/runnerd-restart-continuation.test.ts`. It needs the real Rust `runnerd` binary, and this sandbox has no Rust toolchain. CI builds the runner and runs these tests. - Live acceptance: after the automatic staging deploy, a chat follow-up whose `runnerd` restarts mid-turn must show one continuation turn in the same sandbox instead of a failed run. ## Risks - **Second executor.** The continuation starts only for the turn id that this execution received from its own `startTurn` call. A turn from an earlier process, or a turn that the execution adopted, is not this execution's turn. That path still needs the control-plane restart claim. The continuation runs at most once per run (`restartContinuationStarted`). - **Disk still full.** If the continuation fills the disk again, the second lost turn finds `restartContinuationStarted` set. The run then fails as `sandbox_disk_full`, and automatic retries stay off (#22239). The worst case is the current behavior. - **Disposition-only turns.** A recovered session that only asks for the disposition does not change. This change does not add a continuation for that turn. - Low risk otherwise. The change is 1 condition and 1 assignment in the runner runtime. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, Paperclip API and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (except one test that needs the Rust runner binary; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: the runtime comment states the rule) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: paperclipbugbot <356415228+paperclipbugbot@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5371c83152 |
fix(runner): bundle the Copilot metadata probe as a standalone entrypoint (#22436)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner ships a provider pack with pinned provider runtimes and small entrypoints > - Most provider entrypoints, such as the Kimi metadata probe and the ACPX sidecar, are esbuild bundles that run as one file > - The Copilot metadata probe is plain `tsc` output that imports sibling driver modules > - So an environment that installs only selected provider-pack files cannot run the Copilot probe > - This pull request makes the Copilot metadata probe a standalone verified entrypoint, like the Kimi probe > - The benefit is that every metadata probe has the same self-contained shape, and a consumer can ship the probe as one file ## Linked Issues or Issue Description No public issue exists. This is an enhancement. **What existing behavior does this improve?** The Copilot metadata probe entrypoint (`dist/cli/copilot-metadata-probe.js`, the `paperclip-runner-copilot-metadata-probe` bin and provider-pack shim). **Subsystem affected** Paperclip Runner provider pack build (`packages/paperclip-runner`). **Current behavior** The probe is plain `tsc` output. It imports `../drivers/copilot-metadata-probe.js` and more modules. It fails when a consumer installs only selected provider-pack artifacts. The Kimi probe does not have this problem because `scripts/build-verified-provider-entrypoints.mjs` bundles it. **Proposed behavior** The build bundles the Copilot probe into one self-contained file, like the Kimi probe. The bundle still finds the pinned Copilot assets when it runs from `dist/cli/`. **Reason and benefit** Every metadata probe has the same standalone shape. A consumer can ship the probe as one file. An alternative was to ship the full `dist/drivers` tree with the probe, but that keeps the multi-file shape. **Breaking changes** None. The bin name, the output path and the probe output do not change. ## What Changed - `scripts/build-verified-provider-entrypoints.mjs`: add `copilot-metadata-probe` (source `src/cli/copilot-metadata-probe.ts`, outputs `dist/cli/copilot-metadata-probe.js` and `.cjs`) with the same format and flags as `kimi-metadata-probe`. The bundler asserts one output and only `node:` imports. - `src/cli/copilot-metadata-probe.ts`: move the top-level `await` into `main()`. esbuild cannot emit top-level `await` in the CommonJS bundle. Output and exit codes do not change. - `src/drivers/acpx/provider-assets-root.ts`: when no package root is bound, accept the bundled `dist/cli/copilot-metadata-probe.(js|cjs)` module URL for the `copilot` provider only. This is the same rule the bundled ACPX sidecar uses. Before this change, the bundle's `import.meta.url` is the `dist/cli` file, not `dist/drivers/acpx/copilot-installation.js`, so the module-URL lookup failed. The bound `PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT` path does not change. - Tests: a bundler test that runs both bundles alone in a temporary pack layout, and an asset-root test for the bundled probe path that also proves other providers do not get the new path. ## Verification - `node --test scripts/build-verified-provider-entrypoints.test.mjs` (44 tests, includes the new standalone-run test) - `pnpm exec vitest run src/drivers/acpx/provider-assets-root.test.ts src/drivers/copilot-metadata-probe.test.ts src/drivers/acpx/copilot-installation.test.ts` - `node --test scripts/kimi-metadata-probe.test.mjs scripts/provider-pack-executable-shims.test.mjs test/acpx-codex-package-contract.test.mjs` - `pnpm exec vitest run server/src/__tests__/copilot-connection-probe.test.ts` - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript` - Manual check on darwin-arm64 with the pinned Copilot closure installed, child process spawn denied by a preload: - Bundled `.js` and `.cjs`, no bound root: installation verification passes and the probe reaches the spawn step. - Same run without the `provider-assets-root.ts` change: `COPILOT_INSTALLATION_INVALID`. - Bundled `.js` with `PAPERCLIP_ACPX_PROVIDER_PACKAGE_ROOT` bound: installation verification passes. - `pnpm verify` requires broader coverage (the bundler is a broad input), so CI supplies it. ## Risks - Low risk. The bin name, output path and shim do not change. The probe output and exit codes do not change. - The new module-URL rule accepts only `/dist/cli/copilot-metadata-probe.(js|cjs)` and only for `copilot`. It resolves to the package that holds `dist/cli`, the same as the sidecar rule. - The build now also writes `dist/cli/copilot-metadata-probe.cjs`. Nothing reads it yet. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic Claude - Model: Claude Opus 5.5 (`claude-opus-5-5`) - Mode: agentic coding with tool use (shell, file edits, test runs) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dce4d47f60 |
feat(plugins): carry Grok, Copilot and OpenCode credentials to plugin Runners (#22359)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment-driver plugins can run `paperclip_runner` tasks on hosts
that the plugin owns
> - Core transfers the selected managed AI credential to the plugin as a
typed task credential
> - The credential contract has only two types: a Codex auth file and a
direct Claude credential
> - Grok, GitHub Copilot and OpenCode agents in a task-capable plugin
environment therefore fail closed, although the Runner already reads
their credentials from its environment
> - This pull request adds one generic, allowlisted environment
credential for these three harnesses and accepts the Runner's Grok
launch settings
> - The benefit is that plugin-provided Runner execution can run Grok,
Copilot and OpenCode agents with their selected connection
## Linked Issues or Issue Description
**Subsystem affected**
Plugin environment tasks (`packages/plugins/sdk` task contract) and
plugin-provided Runner execution in `server`.
**Problem or motivation**
An agent with the Grok, Copilot or OpenCode harness cannot run in a
plugin environment that runs Runner tasks. Core stops at credential
preparation with "Plugin Runner does not support the selected credential
provider". A Grok agent fails before that point too. The Runner emits
`grokEffort` and `grokDisableWebSearch` in its launch requirements (Core
turns off Grok web search by default), but the strict `settings` schema
of `environmentRunnerLaunchRequirementsSchema` rejects both keys.
**Proposed solution**
Core transfers the selected connection's credential variables for these
harnesses through one typed `provider-env` credential. Each provider has
a fixed allowlist of complete variable sets. The plugin sets the
variables in the Runner process environment.
**Alternatives considered**
One credential type per harness (`grok-env`, `copilot-env`,
`opencode-env`). This adds a type for each new harness. One generic type
with an exported allowlist keeps the same strictness with less surface.
**Roadmap alignment**
The roadmap lists Grok and OpenCode as supported harnesses. This change
lets plugin environments run them. It adds no new product concept.
**Additional context**
Grok subscription login (an auth file with refresh and writeback) is out
of scope. It still fails closed, with an error that names the xAI API
key as the supported option.
## What Changed
- `packages/plugins/sdk`: Added the `provider-env` task credential: `{
type: "provider-env", provider: "grok" | "copilot" | "opencode",
environment }`. The environment must contain exactly one complete
variable set for the provider and no other variable. Values are
non-empty, NUL-free and at most 65,536 characters, as for the other
credential types.
- `grok`: `XAI_API_KEY`
- `copilot`: `COPILOT_GITHUB_TOKEN`
- `opencode`: `OPENROUTER_API_KEY`, or `PAPERCLIP_AI_PROVIDER_URL` with
`PAPERCLIP_AI_PROVIDER_KEY`
- `packages/plugins/sdk`: Exported the allowlist as
`ENVIRONMENT_TASK_PROVIDER_CREDENTIAL_KEY_SETS` and the
`PluginEnvironmentTaskCredentialProvider` type. `claude-env` and
`codex-auth-json` do not change.
- `packages/plugins/sdk`: Added `grokEffort` (`minimal` to `xhigh`) and
`grokDisableWebSearch` (boolean) to the launch requirements `settings`,
with the types that the Runner uses.
- `server`: `preparePluginRunnerCredentials` now transfers the selected
connection's variables for the `grok`, `copilot` and `opencode`
harnesses, with the same account identity semantics as the Claude
branch. It refuses a Grok subscription, a Grok base URL override, and an
incomplete or ambiguous set. The error never contains a credential
value.
- `server`: The submit check now matches a `provider-env` credential to
the task harness. Before, it treated every non-Codex credential as
Anthropic.
- Drift test:
`server/src/__tests__/plugin-runner-credential-names.test.ts` checks the
SDK key sets against the Runner. The SDK list stays written by hand, and
the Runner keeps its own lists of forwarded names. For `grok` and
`copilot`, each name must be in `ACPX_CREDENTIAL_NAMES`, survive sidecar
admission with the session binding that the Runner mints, and reach the
agent process. The binding requirement must match the Rust sidecar's
candidate agents. For `opencode`, each name must be in the OpenCode
proxy environment set. Each name must also be in the Rust sidecar
credential keys or the Rust OpenCode provider keys. The test reads those
from the Runner source, as the Codex security test does. A provider
without a Runner route fails the test.
`OPEN_CODE_RUNNER_ENVIRONMENT_KEYS` is now exported for this test.
- Docs: Added a "Typed credentials" section to
`doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`. It also documents the ACPX
credential binding that the Runner requires with a Copilot token.
Updated `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`.
## Verification
- `npx vitest run tests/environment-task-release-resume.test.ts
tests/environment-tasks.test.ts` in `packages/plugins/sdk`: 96 passed.
The tests accept each valid shape. They reject mixed sets, two sets,
extra variables, another provider's variable, empty values, NUL bytes,
oversized values, unknown providers and extra fields, on `submit` and on
`resume`. They also check that every published allowlist set parses.
- `npx vitest run
src/services/native-runtime/plugin-runner-credentials.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 79 passed.
The new tests cover each provider's transfer, Grok subscription refusal,
Grok routing refusal, the fail-closed cases without value leaks, harness
matching at submit, and Runner-built Grok requirements
(`runnerProviderLaunchRequirements`) that parse through the task
contract and produce a Grok launch.
- `npx vitest run src/__tests__/plugin-runner-credential-names.test.ts`
in `server`: 4 passed. I broke each list locally to confirm that the
test fails: a renamed SDK key, an SDK provider without a Runner route,
and a dropped name in the Rust sidecar keys, the Rust candidate agents,
the Rust OpenCode keys and the TypeScript OpenCode keys. Each change
failed the test.
- `node scripts/typecheck-layers.mjs` in `server` (includes
`src/__tests__`): no errors.
- `npx vitest run
server/src/services/native-runtime/plugin-runner-probe.test.ts` (from
the root): 15 passed.
`server/src/services/native-runtime/plugin-runner-task.test.ts`: passed.
- `tsc --noEmit` for `packages/plugins/sdk` and `server`: no errors.
- I did not run the full suite locally. CI supplies that coverage.
## Risks
- Low risk for existing plugins. The change only adds a credential type
and two optional settings. A plugin that does not handle `provider-env`
gets a typed value that it does not recognize. Before this change, Core
failed these runs before it called the plugin.
- A plugin must set `PAPERCLIP_ACPX_CREDENTIAL_BINDING` for a Copilot
task. Otherwise the Runner refuses the token. The new doc section shows
the value.
- An OpenCode gateway connection without a key (`auth: none`) still
fails closed, because every value must be non-empty.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, through Claude Code
with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
575ef12575 |
feat(responsibilities): file company skills under responsibilities (#22275)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A responsibility shows the skills of the agents doing its work. People cannot say which skills a responsibility needs. > - The maintainer chose to file skills under responsibilities like projects, tasks and routines. > - Agents also need to set a skill's catalog details without rewriting its SKILL.md. > - This pull request lets people and agents file, move and remove company skills under a responsibility, and adds `update_skill_details`. > - The benefit is that a responsibility lists the skills it needs, beside the skills its agents already hold. ## Linked Issues or Issue Description Refs #22253, #22261 and #22274; this pull request is stacked on them. **Subsystem affected** Database (`responsibility_links`), shared types and validators, server responsibility service and routes, Dashboard versions, native runner tools, and the responsibility page UI. **Problem or motivation** Skills under a responsibility are only computed from the agents doing its work. A person cannot record that "Hiring needs interviewing", and an agent cannot file a skill when it maps work for a person. **Proposed solution** Add a nullable `skill_id` to `responsibility_links` and allow it as the one target of a link. Show filed and computed skills together. Let the file, unfile, lookup and link routes and tools take skills. Keep filed skills in Dashboard versions so undo covers them. Add a small live tool for a skill's tagline and categories. **Alternatives considered** Keeping skills computed only, and giving the skill to the agent that runs the responsibility, was the other option. It was not chosen. Extending `update_skill` would change its scenario contract, whose mock needs `markdown`, so the details tool is separate. ## What Changed - Review fixes: the default responsibility view's Skills "+" adds an existing skill, removes a filed one, or opens an agent's skills page. A newest Dashboard version without a skill list is upgraded in place, so the first skill filed after it is undone. Undo files again, or removes, only skills the restoring person can see. A read scope without a skill condition sees no filed skill. `update_skill_details` with only `skillId` is a no-op, as its schema allows. - Migration `0499`: add `responsibility_links.skill_id` (FK to `company_skills`, on delete cascade). Widen the single-target check to four targets. Add a unique `(responsibility_id, skill_id)` index and a `skill_id` index. The migration is idempotent and online-compatible (NOT VALID then VALIDATE, and the indexes are built CONCURRENTLY). It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. - Shared: `ResponsibilityLink.skillId`; `ResponsibilitySkillSummary.linkId` and `computed`; `skillId` in the link and lookup schemas; `skill` in `RESPONSIBILITY_FILE_ITEM_KINDS`; optional `skillIds` in Dashboard snapshot links. - Server: filing checks the reader's skill visibility (`skillAccessService`). The detail merges filed and computed skills into one card each. Lookup accepts `skillId`. Dashboard capture, restore and summary include filed skills. A version without `skillIds` leaves filed skills as they are when it is restored. - Tools: `file_under_responsibility`, `remove_from_responsibility` and `find_responsibilities_for` accept `kind: "skill"`. `get_responsibility` returns `linkId` and `computed` for each skill. Add `update_skill_details` (tagline and categories). - UI: the responsibility page has "Add existing skill". A filed skill shows "Filed here" and a remove button. Computed skills keep their holders. ## Verification - Server: the `responsibilit*` tests (new: file, find, move and unfile a skill; one card for a skill that is both filed and computed; undo of a filed skill from the Dashboard history), `skill-tools.test.ts` (new `update_skill_details` cases), `responsibility-tools`, `routine-tools`, `routines-routes`, `paperclip-runner-tool-authority`, `runner-api` and `openapi-routes`: 24 files, 2723 tests passed. - UI: responsibility page and component tests (16 files, 158 tests, including a new filed-skill test) pass. `pnpm --filter @paperclipai/ui typecheck` passed. - `@paperclipai/db` `check:migrations` passes and reports the override. The snapshot drift test passes. `tsc` passes for `db`, `shared` and the server layers. - Runner: catalog and semantic-tool tests (317) pass. All generator checks pass. ## Risks - The migration changes a check constraint on `responsibility_links`. The replacement accepts every row that the old check accepted. It is swapped as NOT VALID and then validated. The previous release never sets `skill_id`. - Deleting a company skill removes its links (cascade). - Filed skills follow the reader's skill visibility. A skill the reader may not see is not listed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
b7319eb581 |
feat(routines): let agents list and change their person's routines in plain terms (#22274)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines are recurring work that an agent runs on a schedule, webhook or request. People file them under their responsibilities. > - #22253 and #22261 let agents read and change areas and responsibilities. Agents still have no dedicated routine tools, and they can change only routines assigned to themselves. > - When a person says "move my weekly digest to Monday mornings" or "map my routines to my responsibilities", the agent must use the generic API, write cron, and it is refused for the person's routines. > - This pull request adds `list_routines` and `update_routine`. It also lets an agent change routines that the person its run works for is responsible for. > - The benefit is that an agent can find routines under no responsibility, file them, and change their schedule in plain terms. Each change is a routine revision the person can restore. ## Linked Issues or Issue Description Refs #22261 (phase 2; this pull request is stacked on it), #22253 (phase 1). **Subsystem affected** Server (routine routes, native runner tool authority), the native runner tool catalog, and the `paperclip` skill. **Problem or motivation** An agent can list and change only its own routines through the generic API. It cannot change the schedule of a routine its person is responsible for. It must write cron expressions, and it cannot find the routines that are filed under no responsibility. **Proposed solution** Make the existing scenario operation `list_routines` live, with filters for words, status, agent, project, responsible person, a responsibility and "unfiled". Add `update_routine` for the title, instructions, agent, project, status, priority, a plain `schedule`, and filing under a responsibility. Let an agent manage a routine when the person its run works for is responsible for it. Handing the routine to another agent needs that person to have `tasks:assign`. **Alternatives considered** A new `routines` permission grant would add a setting with no current need. Routine revisions already give an undo path, so this change adds no new history. ## What Changed - `list_routines` is now live (it was a scenario-only mock) and keeps its scenario mock. - Add `update_routine`. A `schedule` such as `{ frequency: "weekly", dayOfWeek: "monday", time: "09:00", timezone }` maps to the routine's schedule trigger through the existing routine-offer cadence helpers. `schedule: null` turns the schedule off. `fileUnder` files the routine under a responsibility. - Add `server/src/services/routine-tools.ts`. It calls the routine routes with the run token. Fields and schedule go in one `PATCH /routines/:id` (one transaction and revision, guarded by `baseRevisionId`); filing follows, and if it fails after the routine changed the tool reports what was saved. Each part reports its own undo: the routine revision, or the Dashboard history for filing. - `PATCH /routines/:id` takes `schedule` (`{ cronExpression, timezone? }` or `null`) and replaces the schedule under the routine row lock, so racing changes leave one schedule. Without a zone it keeps the current schedule's, else uses the person's saved time zone, else answers 422. - `GET /companies/:companyId/routines` filters `q`, `status`, `assigneeAgentId`, `responsibleUserId`, `responsibilityId` and `unfiled=true` in SQL. - In `routes/routines.ts`, an agent may manage routines assigned to itself or routines its person is responsible for. On a routine the agent does not run, taking it from another agent (or handing it on), turning it on, changing its schedule (also through the trigger routes), starting a run, or restoring a revision requires that person's `tasks:assign`, as for a person. The person must have directed the run (see #22261). `routine.updated` activity records `onBehalfOfUserId`. - The routine and responsibility routes share one rule, `agentPersonInCompany` in `routes/agent-person-authz.ts`. - The catalog keeps each live action's `requiredClaims` (`list_routines` is `routines:read`). - The routine tools stay visible when Responsibilities is off. Only `responsibilityId`, `unfiled` and `fileUnder` need it. - Update `search_api` tool pointers, the routines reference, the skill section, the chat directive, the operation-groups spec and the pinned counts. ## Verification - Server: `routine-tools.test.ts` (new), `routines-routes.test.ts` (new: an agent changes its person's routine, hands it off only with `tasks:assign`, and is refused for another person's routine), the `responsibilit*` tests, `paperclip-runner-tool-authority` and `runner-api`: all pass. The first test in `routines-routes.test.ts` (Fireflies) times out at 15 s in my sandbox on a cold import, with or without this change. It passes with `--testTimeout=60000`. - Server `node scripts/typecheck-layers.mjs`: passed. - Runner: catalog, semantic-tool, eval, scenario and permission-policy tests (316) pass. All generator checks pass. ## Risks - Medium-low. Agents get write access to their person's routines. The route still checks the company, the routine revision guard and `tasks:assign` for a hand-off. The person can restore any earlier routine revision. - `list_routines` changes from a scenario-only mock to a live tool, so the reconciled catalog counts change. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
c0aa7751f8 |
feat(responsibilities): let agents change areas and responsibilities for their person, with undo (#22261)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People keep their areas and responsibilities in Paperclip (experimental, `enableResponsibilities`). #22253 lets agents read them. > - Agents can still only propose. They cannot rename or archive an area, change an active responsibility, let one go, or move work between responsibilities in one step. > - A person who says "file my weekly routines under Sales" must then do it by hand. > - The maintainer chose this rule: an agent may make each change that its person could make, the person can undo it from the Dashboard history, and a permanent delete still needs the person's yes on a card. > - This pull request adds eight change tools and the server rules behind them. > - The benefit is that an agent maps work to responsibilities for the person, and the person keeps control through undo and the delete card. ## Linked Issues or Issue Description Refs #22253 (phase 1; this pull request is stacked on it). **Subsystem affected** Server (responsibility and area routes, Dashboard versions, native runner tool authority), shared validators and Dashboard types, the native runner tool catalog, the `paperclip` skill, and the Dashboard history UI. **Problem or motivation** An agent that works for a person can only propose areas and responsibilities. It cannot change active ones, archive or delete them, or move filed work in one step. Unfiling needs link IDs. There is no way for the person to undo an agent's change. **Proposed solution** When an agent acts for the person its run works for (`onBehalfOfUserId`) and that person directed the run (the run's own task is one they made, such as their chat, or their own message woke it; parent tasks do not count), apply that person's own rights: their own responsibilities, or anyone's for a company owner or admin; areas for any active member. Save each such change as a version of the person's Dashboard, so the existing history and Undo line show it under the agent's name. Make a permanent delete wait for an accepted confirmation card that the agent posted. Add `file` and `unfile` routes that take items by kind and ID, with an optional move. **Alternatives considered** One card for every change, or propose-only. The maintainer chose "act for me, with undo". A separate undo journal would need a new table and a new UI; Dashboard versions already have both. A new version cause needs a check-constraint migration; this change uses `change_set` with no `interactionId` instead. ## What Changed - Add tools `create_area`, `update_area`, `delete_area`, `create_responsibility`, `update_responsibility`, `delete_responsibility`, `file_under_responsibility` and `remove_from_responsibility`. They are standard-mode only and hidden while Responsibilities is off. - Add the writer rule for agents in `routes/agent-person-authz.ts` (`agentPersonInCompany`, `agentActsForResponsibilityOwner`), used by `routes/responsibilities.ts` and `routes/responsibility-areas.ts`. The person must have directed the run: a person can name anyone as a task's responsible person, and a run can also work for a routine's person or the company default, so those runs do not borrow that person's rights. An agent with no such person keeps the old propose-only rules. - The response names only the version of the responsibility's owner (for area changes, the acting person's), and `X-Paperclip-Dashboard-Owner` says whose history holds it; tools return `dashboardOwnerUserId`. A change Undo cannot reverse (the name, description, order or un-archiving of an area other people's proposed or active responsibilities share) saves no version and is not reported as undoable. - `/file` without `moveFrom` follows the `/links` rule, so `create_responsibility` with `fileItems` works for a proposal. - A title-only create is allowed for an acting agent. It is active by default and records the agent in `proposedByAgentId`. - Add `responsibility-agent-changes.ts`. In one transaction that holds each affected owner's Dashboard lock, it saves the dashboard before the change (when a person changed it by hand), runs the change on services bound to that transaction, and saves the result as a `change_set` version by that agent. The response names the version in `X-Paperclip-Dashboard-Version`. The tools say Undo only when a version was saved. - Dashboard snapshots now also hold each responsibility's filed items, description, "run by" agent, evidence, health override and note, and each area's description (optional fields). Restore puts them back. Older versions still restore as before. A matching older version is upgraded in place, so the first change after it compares and undoes in full. - Create and update accept `areaTitle`. The server finds or makes that area in the same transaction, so a refused change leaves no area. Agent creates reuse a live responsibility with the same title under a lock, so retries and parallel calls make no copies. - Add `POST /api/responsibilities/:id/file` (`items`, optional `moveFrom`: an ID or `all`) and `POST /api/responsibilities/:id/unfile` (`items`), each in one transaction. - Agent deletes need `confirmationId`: an accepted `request_confirmation` card that this agent posted, with target key `responsibility-delete:<id>` or `responsibility-area-delete:<id>`. Someone who may manage the owner must accept it, on the card or by a verified chat reply. The tools post the card (named after the item), return `awaiting_confirmation`, and delete on the next call after acceptance. - `DELETE /api/responsibility-areas/:id` accepts `moveResponsibilitiesTo`. Area changes and deletes save versions for everyone whose Dashboard shows the area. - Agent activity details now include `onBehalfOfUserId`. The Dashboard history shows "<agent>, for you" for these versions. `ResponsibilityDashboardVersionSummary` now has `interactionId`. - Update the OpenAPI registry and metadata, the `search_api` tool pointers, the skill, the API reference, the chat directive, the operation-groups spec, and the pinned counts and SDK source pins. ## Verification - Server: `vitest run` of the `responsibilit*` tests, `responsibility-tools.test.ts`, `openapi-routes`, `runner-api*`, `paperclip-runner-tool-authority` and `task-markdown` all pass. New tests cover these cases: acting-agent create, update and undo; file, move and undo; the rule for a person who may not manage the owner; delete only after an accepted card, including a verified chat reply; area delete with move and a version for the other owner; undo of a health override, an area description and an un-archived empty area; upgrade of an older version; an area by title that a refused change rolls back; parallel creates. - UI: `DashboardEditModule.test.tsx` and `responsibility-dashboard.test.ts` passed. `pnpm --filter @paperclipai/ui typecheck` passed. - Server `node scripts/typecheck-layers.mjs` passed. - Runner: catalog, semantic-tool, eval, protocol-action and permission-policy tests (284) passed. All generator checks pass. - I did not run the browser e2e suite locally. ## Risks - Medium. Agents that work for a person can now change and archive that person's responsibilities and areas without a card. Every change is a Dashboard version that the person can restore, and a delete still needs the card. - Undo restores titles, sentences, areas, order, status, filed items, description and the "run by" agent. It does not restore a bar or metric change; those stay in the metric history. - A restore of a version from before this change does not touch filed items, which matches the earlier behavior. - Undo of a rename or archive of an area that other people also use keeps its name, as the existing restore rule does for shared areas. - Manual edits do not take the Dashboard lock. A person's edit that commits while an agent's change is open can appear in that agent's version. - Changes to proposed or let-go responsibilities are not on a Dashboard, so they cannot be undone there. The tools say so. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ffcd49f6aa |
feat(responsibilities): let agents read a person's areas and responsibilities (#22253)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People keep their areas and responsibilities in Paperclip (experimental, `enableResponsibilities`). Agents chat with them about this work. > - Agents have no dedicated tool to read a person's responsibilities. The only path is the generic `search_api` and `call_api` fallback. > - That path needs the person's user ID, and nothing in a chat tells the agent that responsibilities exist. When the feature is off, every route returns a 404 that looks like "nothing there". > - So a chief-of-staff agent asked the person "what are your responsibilities, or where are they written down?" when Paperclip already had them. > - This pull request adds three read tools, makes the owner default to the person the run works for, and tells agents in chats how to use the tools or that the feature is off. > - The benefit is that an agent reads and maps a person's responsibilities itself. This is phase 1 of 4. Later pull requests add the change tools for areas, responsibilities, routines and skills. ## Linked Issues or Issue Description **Subsystem affected** Server (responsibility routes, native runner tool authority, chat prompt), native runner tool catalog, and the `paperclip` skill. **Problem or motivation** An agent in a chat cannot find a person's responsibilities. It has no dedicated tool. The list route requires `ownerUserId` from an agent, and the chat does not give the agent that ID. The chat prompt does not mention responsibilities. When the experiment is off, the 404 does not tell the agent that the feature is off. The agent then asks the person to list their responsibilities again. **Proposed solution** Add three optional native tools: `list_responsibilities`, `get_responsibility` and `find_responsibilities_for`. They call the existing routes with the run's agent token, so all route checks and the activity log stay the same. The owner defaults to the person the run works for. The routes return a `responsibilities_disabled` code while the feature is off. The tools are hidden while the feature is off, and the chat prompt then says that the feature is off. **Alternatives considered** Better `search_api` guidance alone still needs the user ID and many calls. A new read service in the tool authority would copy the route permission rules. The tools use the routes instead. ## What Changed - Add `list_responsibilities` (areas with their responsibilities, responsibilities with no area, totals, and company work filed under no responsibility), `get_responsibility` (one responsibility with its filed items, link IDs, skills and latest status report), and `find_responsibilities_for` (where a project, task or routine is filed; a task can be a `PAP-12` identifier). - Add `server/src/services/responsibility-tools.ts`. It calls the routes over HTTP with the run token and returns compact results with page links. - When an agent sends no `ownerUserId`, the responsibility list, create, summary, automation and dashboard routes use the person the run works for (`onBehalfOfUserId`). A run that works for no person still gets a 422 with a clear message. - The nine "not enabled" checks now throw one shared error. The 404 and its message do not change. The body now also has `code: "responsibilities_disabled"` and a remediation line. One `RESPONSIBILITIES_OFF_GUIDANCE` text is the remediation, the tool error and the chat prompt note. - `list_responsibilities` with `areaId` counts only that area. Archived areas are listed only on request. When only proposals are hidden, the tool says so. - `services/paperclip-api-loopback.ts` holds the tools' authenticated API call; the project tools use it too. - Show the tools only while the instance has Responsibilities on. A hidden tool that is called by name returns "Areas and responsibilities are turned off". - In standing chats, add an "Areas and responsibilities" prompt block: use the tools before you ask the person, or say that the feature is off. - `search_api` results for these routes now name the dedicated tools. - Add an "Areas and responsibilities" section to `skills/paperclip/SKILL.md` and to `references/api-reference.md`. Add one line to the chief-of-staff persona. - Register the tools in the runner catalogs. Regenerate the semantic contracts, the action catalog, protocol coverage, the operation-groups spec and the capability inventory. Update the pinned counts. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/catalog src/semantic-tools src/tools src/eval/workflow-evals.test.ts src/protocol-actions src/evals/native-execution.test.ts src/drivers/acpx/permission-policy.test.ts`: 276 passed. - All runner generator checks pass: protocol manifest, capability contract, capability inventory, protocol coverage, semantic contracts, semantic action catalog, operation groups, replay goldens, workflow traceability. - Server: `vitest run` of `responsibilit*` route tests, `responsibility-tools.test.ts` (new), `task-markdown.test.ts`, `task-context*`, `paperclip-runner-tool-authority.test.ts`, `runner-api.test.ts`, onboarding first-task tests and `user-facing-wording-guidance.test.ts`: 20 files, 2620 tests passed. - `node scripts/typecheck-layers.mjs` for the server: passed. - New tests cover these items: the owner default for an agent and the 422 without a person, the `responsibilities_disabled` code on every route, the tool projections, an identifier lookup, the "turned off" mapping, the tool visibility switch, and the chat prompt block. ## Risks - Low risk. The routes keep their permission checks. Agents can still only read and propose in this phase. - An agent that sends no `ownerUserId` now gets the responsible person's list instead of a 422. Agents that name an owner see no change. - The chat prompt gets one more short block in standing chats. - The flag is pinned when a turn opens. If an admin changes it during a turn, the tool list changes on the next turn. The routes always check the current setting. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1c67687998 | feat(runner): start a released Runner again as the next run (#22288) | ||
|
|
170e1b0c07 |
feat(runner): pin Codex 0.170.0 and Agent SDK 0.3.328 so Fable 5.2 and GPT-6.2 run (#22278)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude and Codex agents from one shared provider pack. The pack pins an exact Claude Agent SDK (which bundles Claude Code) and an exact Codex CLI. > - #22222 added Claude Fable 5.2 (minimum Claude Code 2.1.321) and GPT-6.2 Sol and Astra (minimum Codex 0.169.1). The pack still bundles Claude Code 2.1.286 and Codex 0.160.0. > - Because of this, Paperclip Runner stops these models with a named version error, and native remote Codex accepts only `>=0.149.0 <0.161.0`. > - This pull request moves the provider pack to Agent SDK 0.3.328 (Claude Code 2.1.328) and Codex 0.170.0. It moves every constant that depends on these pins, and the remote Codex window to `<0.171.0`. > - The benefit is that Fable 5.2 and GPT-6.2 run on Paperclip Runner after a sandbox image with this pack is promoted. ## Linked Issues or Issue Description Refs #22222 (adds the models and the version floors; this pull request makes them runnable on Paperclip Runner). Refs #18488 (moves only the Codex pin to 0.166.0 and the window to `<0.167.0`; it is not updated since November 1 and has conflicts with `master`). Refs #18486 (GPT-6.1 Terra needs Codex 0.165.0; Codex 0.170.0 satisfies it). **Overlap with #18488.** #18488 changes the same Codex pin locations. This pull request uses a newer pin (0.170.0) and also moves the Claude runtime. If this pull request merges first, #18488 has no remaining change. I did not change the #18488 branch. ## What Changed - Claude runtime: `@anthropic-ai/claude-agent-sdk` 0.3.286 → 0.3.328 (Claude Code 2.1.286 → 2.1.328). This changes the root `pnpm.overrides`, `pnpm-workspace.yaml`, the ACPX `claude` profile, and the qualified Claude runtime in `installation-integrity.ts`, with new executable digests for Linux x64, Linux ARM64, macOS ARM64, and macOS x64. - Codex runtime: `@openai/codex` 0.160.0 → 0.170.0. This changes the runner dependency, the `codex-acp@1.6.2>@openai/codex` override, the `codex-acp` patch dependency line, the ACPX `codex` profile, and the qualified Linux x64 runtime with its new executable digest. - The ACP bridges do not change: `claude-agent-acp` 0.73.0 and the patched `codex-acp` 1.6.2. Their `commandDigest` values and profile versions do not change. - Generated declarations: `generated-profiles.ts` and `generated_acpx_profiles.rs` are regenerated with `generate:acpx-profiles`. - Remote native Codex window: `REMOTE_CODEX_MAXIMUM_VERSION_EXCLUSIVE` 0.161.0 → 0.171.0. The minimum stays at 0.149.0. Images with Codex 0.149.0 through 0.170.x are accepted. - Codex company observability: the qualified-version check moves to `codex-cli 0.170.0`. The `otel` configuration keys that Paperclip writes (`exporter`, `trace_exporter`, `metrics_exporter`) are present in the 0.170.0 configuration schema. The `OtelExporterKind`, `OtelHttpProtocol`, and `OtelTlsConfig` definitions are identical to 0.160.0. - Daytona runner image probes (`Dockerfile`, `Dockerfile.release`) expect `codex-cli 0.170.0` and `2.1.328 (Claude Code)`. The remote npm install spec in `server/src/adapters/registry.ts` is `@openai/codex@0.170.0`. - Tests: fixtures and assertions that mirror the pins use the new versions. The window tests accept 0.161.0 through 0.170.99 and reject 0.171.0. The ACPX gate tests now show that Fable 5.2 and GPT-6.2 are admitted. A new test mocks an older bundled runtime and keeps the named version-gap errors covered. The fallback test shows that GPT-6.2 stays on 0.170.0 and still falls back on 0.160.0. - Opt-in qualification scripts (`scripts/tests/native-cleanup-paginated-codex.mjs`, `native-provider-startup-codex.mjs`) require Codex 0.170.0. - Documentation: `docker/daytona-runner/README.md`, `doc/architecture/paperclip-runner-compatibility.md`, `doc/agent-observability.md`, `docs/deploy/environment-variables.md`, and the deferred item in `doc/adapter-model-audit-2026-11-18.md`. - `pnpm-lock.yaml` is not part of this pull request. The refresh bot regenerates it on `master`, as for the previous pin refreshes (#14942, #14993). ## Verification Sources, checked on 2026-11-19: - npm registry: `@openai/codex@0.170.0` (published 2026-11-18T03:45Z) and `@anthropic-ai/claude-agent-sdk@0.3.328` (published 2026-11-18T23:13Z, `claudeCodeVersion` 2.1.328). Both are the `latest` tags. - [Codex releases](https://github.com/openai/codex/releases) 0.161.0 through 0.170.0. The notes record no app-server protocol break. 0.169.0 keeps project configuration disabled for `codex exec` in untrusted directories; Paperclip's CLI path already uses `--skip-git-repo-check`, and the native path uses `app-server`. - [Claude Code changelog](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md) 2.1.287 through 2.1.328. See "Risks" for the headless changes that I reviewed. Digests and package shape: - I downloaded each platform archive with `npm pack`, which verifies registry integrity, and computed the SHA-256 of the executable. As a method check, the same command on the 0.3.286 Linux x64 archive gives the digest that is on `master` (`fe503f65…`). The Codex 0.160.0 archive also gives the `master` digest (`12eb3e81…`). - The extracted executables report `codex-cli 0.170.0` and `2.1.328 (Claude Code)`. - The Codex 0.170.0 Linux x64 archive no longer contains `codex-resources/zsh/bin/zsh`. Paperclip does not reference that path. - The Agent SDK 0.3.328 `optionalDependencies` have the same platform set as 0.3.286. The Codex 0.170.0 `optionalDependencies` use the same `npm:@openai/codex@<version>-<platform>` aliases. - Bridge link check: I installed `@agentclientprotocol/claude-agent-acp@0.73.0` with an override to Agent SDK 0.3.328 and imported its entry module. Node linked all of the named SDK imports, and the import completed. Real Codex qualification (no model turn, no credentials): - `PAPERCLIP_TEST_CODEX_BINARY=<installed 0.170.0 Linux x64 binary> node --import ./server/node_modules/tsx/dist/loader.mjs scripts/tests/native-cleanup-paginated-codex.mjs` passed: `green: true`, with the expected red control and paginated `thread/resume` through the real 0.170.0 `app-server`. The installed binary digest matches the pinned digest. Focused suites (Linux x64, Node 25.9.0, local `pnpm install --no-frozen-lockfile`): - `cd packages/paperclip-runner && pnpm run typecheck:typescript` passed, including `check:acpx-profiles`. - `npx tsc --noEmit` passed in `packages/adapters/codex-local` and `packages/adapters/claude-local`. - `packages/adapters/codex-local`: 625 passed. `packages/adapters/claude-local`: 419 passed, 1 skipped. - `node --test`: `acpx-codex-package-contract` 9 passed, `controlled-evals/controller` 77 passed, `scripts/acpx-patch-packaging` 26 passed, `scripts/__tests__/grok-public-install-sandbox` 27 passed. - `server`: `acpx-qualification`, `codex-runtime-compatibility`, and `codex-model-fallback` 110 passed. `native-session-executor` 1541 passed, 1 failed. The failure (`compares export process compatibility … db.transaction is not a function`) also fails on unchanged `master` on this host. - `packages/paperclip-runner` vitest (`installation-integrity`, `qualified-profiles`, `codex-security-config`, `native-execution`, `native-backend-factory`): 283 passed, 25 failed. All 25 failures are in `installation-integrity.test.ts` with "ACPX provider lifetime guardian failed to start". The same 25 tests fail on unchanged `master` on this host (master fails 30 here). Rust (toolchain 1.97.1 from `rust-toolchain.toml`, Linux x64): - `cargo test -p paperclip-runner-core --lib`: 658 passed, 2 failed, 4 ignored. `warm_run_attachment_reuses_the_provider_ingress_listener` also fails on unchanged `master` on this host. `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` is flaky here: on this branch it passed 2 of 3 reruns. - `cargo test -p paperclip-runner-core --test acpx_warm_attachment --test native_provider_backend`: 7 and 45 passed. - The new and changed saved-session tests pass: `a_session_saved_by_the_previous_agent_runtime_is_refused_by_name`, `unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class`, and `a_settled_session_saved_by_the_previous_acpx_release_continues`. `rustfmt --check` passes. - `server`: `native-session-resume.test.ts` and `acpx-qualification.test.ts` 110 passed, 2 skipped. Both new resume cases run and pass. Not run here: - `native-provider-startup-codex.mjs`, because it needs a compiled `paperclip-runnerd`. The trusted CI runner workflow covers startup and session-resume probes. - The Docker image build and any live model turn. A live Fable 5.2 and GPT-6.2 test follows the staging image rollout. ## Saved sessions from the previous release This change keeps the ACPX bridge command digests and profile versions. It changes `agentRuntimeVersion` in the `claude` and `codex` ACPX profiles. That field is part of the saved session identity. The previous release saved sessions with Agent SDK 0.3.286 or Codex 0.160.0. This build refuses those saved sessions. It does not migrate them. - **Next run (the usual case):** the saved provider descriptor names the old runtime. `rebindNativeSessionCheckpoint` does not match it, so the next run starts a fresh provider session in the same workspace with the complete handoff prompt. No run fails. The earlier provider conversation does not carry over. - **A run in flight at the deploy (settled or mid-turn):** the runner restore refuses the saved descriptor with the permanent class `provider_state_incompatible`. The server reports it as `native_provider_state_incompatible` and stops automatic retries. The runner opens no provider and keeps the saved state bytes and the workspace. The next run on the task starts a fresh provider session. - **Native Codex (`codex_app_server`) and OpenCode** sessions do not record this field, so their resume decision does not change. After a new image is promoted, native Codex resumes a saved thread with Codex 0.170.0 instead of 0.160.0. This pull request does not test that cross-version thread resume. New tests: - `acpx_provider_backend.rs`: `a_session_saved_by_the_previous_agent_runtime_is_refused_by_name` covers settled Claude and Codex sessions and a Claude turn interrupted mid-run by the deploy. `unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class` has a new `agent_runtime_release` case. Each case checks the named class, that no provider session opens, and that the saved bytes do not change. - `native-session-resume.test.ts`: "a session saved by the previous Claude or Codex ACPX runtime" checks that a same-release session resumes, and that a session saved with Agent SDK 0.3.286 or Codex 0.160.0 starts fresh in the same workspace. ## Risks - Medium. Every Paperclip Runner Claude and Codex run uses the new executables. The digests, generated profiles, image probes, and window must agree. They agree locally, and CI runs the Rust and runner suites again. - Claude Code 2.1.287 through 2.1.328 change some headless defaults. I checked each one against Paperclip: - 2.1.302: `-p` and Agent SDK sessions with no permission mode start in `default` mode on third-party providers. Paperclip always sets a mode (`bypassPermissions` or `default`), so this does not apply. - 2.1.299: MCP allow rules now match only the whole server name. Paperclip already writes `mcp__<server>__*` and exact `mcp__paperclip__<tool>` rules. - 2.1.302: fork subagents are on by default in `-p` and the Agent SDK, as in the terminal. This pull request keeps the upstream default and does not set `CLAUDE_CODE_FORK_SUBAGENT=0`. Reviewers can ask to keep them off. - 2.1.299 and later: stdio MCP connections negotiate MCP protocol 2026-07-28 first and fall back to the older protocol for servers that do not answer. - Agent SDK 0.3.328 is less than one day old. If a regression appears, revert this pull request to return to 0.3.286 and Codex 0.160.0. - Sandbox images built before this change still carry Codex 0.160.0 and Claude Code 2.1.286. They stay accepted. On those images, Fable 5.2 and GPT-6.2 keep the existing named errors and fallbacks until a new image is promoted. - Until the refresh bot updates `pnpm-lock.yaml` on `master`, a local `--frozen-lockfile` install fails, as with the previous pin refreshes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, through Claude Code with tool use (shell, file edit, `gh`) and code execution. The context window and reasoning mode are not reported in this session. - The registry archives, digests, release notes, configuration schemas, and changelog were read directly. No model output was used as a source for those facts. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
07d9316218 |
fix: restore master's typecheck and runner SDK input pin (#22242)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #22235 added a runner regression test for a turn lost to a mid-run `runnerd` restore > - That test reads `startTurn.mock` inside the initializer of `startTurn` itself > - The CI test-program typecheck rejects this with TS7022 and TS7024, so master fails "Typecheck + Release Registry" > - This pull request counts calls in a local variable instead > - The benefit is that master typechecks again, and the test behaves exactly as before ## Linked Issues or Issue Description Refs #22235, Refs #22234 **What happened?** After #22235 merged, `ci / Typecheck + Release Registry` failed on `packages/paperclip-runner/src/native-session-runtime.test.ts(724)` with TS7022 ("'startTurn' implicitly has type 'any'...") and TS7024. **Steps to reproduce** 1. Run the CI typecheck lane (`typecheck test programs`) on master after #22235. 2. See TS7022 and TS7024 at line 724 of the test. **Expected behavior** The test program typechecks. ## What Changed - The `startTurn` mock in the new test uses a local call counter instead of reading its own `mock.calls`. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the pin for `packages/shared/src/app-definitions/github.json`, which #22234 changed without updating it. The Runner support suite fails on master without this. ## Verification - `npx tsc -p packages/paperclip-runner/tsconfig.test.json --noEmit` reports no errors in the test file. - `npx vitest run src/native-session-runtime.test.ts` in `packages/paperclip-runner`: 301/301 pass. ## Risks - Low risk. This changes a test and regenerates a pin with the repository script (`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --write`). `source-sdk-public-inputs.test.mjs`: 13/13 pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d668eeb660 |
fix(runner): keep Claude MCP bearer tokens out of process arguments (#22206)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for work
> - The runner starts Claude through ACPX and the Claude ACP agent.
> - The runner gives Claude HTTP MCP servers: the task tool bridge and hosted connectors. Each server has a bearer token.
> - The Claude Agent SDK writes these servers, with their headers, into the inline `--mcp-config` argument of the Claude CLI.
> - Thus a `ps aux` or `ps -ef` command in the sandbox shows the bearer tokens. Tool logs then keep the tokens.
> - This pull request sends a `${VAR}` reference in the header. The token goes into the launch environment of the agent.
> - The benefit is that process lists do not show MCP bearer tokens.
## Linked Issues or Issue Description
**What happened?**
An agent ran `ps aux` in its sandbox. The output showed the Claude CLI process with an `--mcp-config` argument. That argument had an `Authorization: Bearer <token>` header for each HTTP MCP server. The tool log kept the full output.
**What did you expect to happen?**
Process arguments must not contain credentials. A process list must not show MCP bearer tokens.
**Steps to reproduce**
1. Start a Claude agent on the runner with the task tool bridge or a hosted connector.
2. In the agent, run `ps -ef`.
3. Look at the Claude CLI line. The `--mcp-config` JSON contains `"Authorization":"Bearer ..."`.
**Root cause**
`openQualifiedAcpxRuntime` sent the literal `Bearer <token>` header in the ACP `mcpServers` list. The Claude ACP agent gives this list to the Claude Agent SDK. The SDK always serializes it into `--mcp-config <json>` on the CLI command line.
## What Changed
- For the Claude ACPX profile, each HTTP MCP header now has the value `Bearer ${PAPERCLIP_ACPX_MCP_BEARER_<n>}`.
- The runner puts each token in the Claude launch environment under the same name. Claude Code expands `${VAR}` in `--mcp-config` when it connects to the server.
- Other ACPX agents keep the literal header. Their CLIs do not have a known `${VAR}` expansion.
- Added a regression test. Updated one exact launch-environment assertion for the new variable.
## Verification
- Failing before the fix: `npx vitest run src/drivers/acpx/acpx-runtime-adapter.test.ts -t "MCP bearer tokens out of the CLI argv"` (in `packages/paperclip-runner`). It failed with `expected '[{"type":"http","name":"paperclip",…' not to contain 'bridge-secret'`. The projected header had the literal synthetic token.
- Passing after the fix: `npx vitest run src/drivers/acpx/acpx-runtime-adapter` — 7 files, 136 tests passed.
- Manual check of the expansion: I ran the pinned Claude CLI (agent SDK 0.3.286) with `--mcp-config` and a header `Bearer ${SYNTH_MCP_BEARER}` against a local HTTP server. The server got the expanded synthetic value.
- `tsc --noEmit -p packages/paperclip-runner`: no new errors. The only errors are in `src/eval/workflow-harness.ts` and occur on `master` too, because my local install did not include the eval kernel package.
- The full `src/drivers/acpx/` folder has other failures in this sandbox (`EACCES` on `/proc/self/fd` spawn, directory fsync). The same failures occur on `master` without this change.
## Risks
- Medium-low risk. If a future Claude CLI stops the `${VAR}` expansion in `--mcp-config`, MCP calls fail authentication. They do not leak the token. The regression test pins the header shape.
- The token is now in the Claude process environment. Only the same user can read `/proc/<pid>/environ`. Other credentials are already in this environment. Shell tools that Claude starts also inherit the environment, so `env` output can still show the token. Log redaction must continue to cover bearer values.
- Tokens that already appeared in logs are not revoked by this change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge
Related: #13572 removes bearer tokens from curl arguments in the agent helper scripts. This pull request covers the Claude CLI MCP configuration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
06750cb18e |
fix(runner): continue a turn lost to a mid-run runnerd restore (#22235)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a managed sandbox run under the Paperclip runner (`runnerd`). When `runnerd` exits mid-run, the transport relaunches it in the same execution, and the runner restores the same conversation > - An agent can fill the sandbox disk with its own work. `runnerd` then cannot write its state and exits. #22068 added a disk reserve so the relaunch succeeds. Its stated goal was that "the agent's turn can continue and see the full disk" > - The relaunch now works, but the turn still dies. The runner reports the in-flight turn as `provider_turn_lost_on_restore`. The runtime sends its single same-conversation continuation only for a session recovered at startup under a control-plane restart claim. A relaunch inside the live execution has neither, so the run fails as `native_provider_terminal_failed` > - The generic transient retry then starts on a fresh sandbox, redoes the same install, fills the disk again, and ends in board-owned "Recovery needed" > - This pull request lets the execution that started a turn send the existing one-time continuation when the runner reports that same turn lost on restore > - The benefit is that the agent keeps its conversation and workspace and sees the full disk in its own tools, instead of the run dying and retrying into the same wall ## Linked Issues or Issue Description Refs #22068 **What happened?** On staging, two agent tasks on the Paperclip monorepo each failed on every attempt (5 failed runs in total). Each run used a fresh managed Daytona sandbox with 10 GiB of disk. The agent's own `df` showed `/` at 10G of 10G with 236K free. `node_modules` used 3.2G and `.paperclip-runtime` used 6.1G: about 3.0G of provider pack and a 2.8G pnpm store under the session data directory. `runnerd` exited with `failed to create private durable state temporary file: No space left on device (os error 28)`. The transport relaunched it in about 20 s, and runnerd restored its durable session. Then the runner reported `turn.failed {code: provider_turn_lost_on_restore, recoverable: true}`, and the run failed as `native_provider_terminal_failed`. Retries on fresh sandboxes hit the same failure. The tasks ended in "Recovery needed" with `provider_failure_meaning_unverified`. **Steps to reproduce** 1. Run an ACPX or Codex agent through `paperclip_runner` in a managed sandbox. 2. Have the agent fill the disk during a turn, or end `runnerd` in any other way mid-turn while the server keeps running. 3. The transport relaunches `runnerd`, and the runner reports the turn as lost on restore. 4. The run fails instead of continuing the turn. **Expected behavior** The execution that started the turn sends one continuation turn in the restored conversation, exactly as it does after a server restart. The agent continues from its recorded progress and sees the full disk in its tools. **Environment** Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB disk), `paperclip_runner` native runtime with Claude through ACPX. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: the execution records the turn id of the turn it starts. `resumeInterruptedTurn` now accepts two owners of a turn lost on restore: - a session recovered at startup with the control-plane restart claim, for the checkpointed active turn (unchanged), or - a session this execution opened itself, for the turn this execution started (new). - All other guards stay: one continuation per run, the provider allowlist, no compaction or goal runs, no pending runtime requests, and the snapshot must show that same turn as restart-interrupted. - `native-session-runtime.test.ts`: a new regression for an ACPX session that loses its own turn to a mid-execution `runnerd` restore, plus a negative case for a lost turn this execution never started. ## Verification - Red before the fix: `npx vitest run src/native-session-runtime.test.ts` failed `continues its own turn once when runnerd is restored mid-execution` with `NativeProviderTerminalFailure: Provider turn failed`, the same failure seen on staging. - Green after the fix: `native-session-runtime.test.ts` and `native-session-runtime.provider-turn.test.ts` pass (337/337). The existing `requires the control plane restart claim` case still refuses a recovered session without the claim. - `pnpm verify` (affected selection): see the CI results on this PR. - Live acceptance: after deploy, retry the affected staging tasks and confirm that a mid-turn `runnerd` restore leads to one continuation turn in the same sandbox instead of `native_provider_terminal_failed`. ## Risks - **Second executor.** The continuation can start only from the execution that opened the session and started the turn. It is not used for a recovered or adopted session without a claim. It runs at most once per run (`restartContinuationStarted`). The runner emits this code only after it proves the old provider process ended. - **Run starts.** This starts no new runs. It adds at most one turn to a run that would otherwise fail and schedule a fresh-sandbox retry. During a crash loop, `MAX_UNSTABLE_RUNNER_RESTORES` in the transport still fails the transport. - **Disk still full.** If the continuation fills the disk again, the second lost turn finds `restartContinuationStarted` set, and the run fails as it did before. The worst case is the current behavior. - **Not in this PR.** Paperclip's own footprint (about 3G of provider pack per sandbox, and a per-session pnpm store) and classifying a disk-full failure so it does not retry into the same wall. Both are follow-ups. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser, and a code-tracing subagent). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ee68098584 |
fix(runner): keep current task requests readable in OpenCode traces (#21584)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Company traces let operators compare each request with its reply. > - OpenCode limits each text field to 8 KiB. > - A fresh session can put old history before the current request. > - The limit can remove the current request from the trace. > - This change selects the current task text first and links the retained submission. ## Linked Issues or Issue Description Refs #18098 and #18699. Neither merged change selects the current task request. **What existing behavior does this improve?** OpenCode company traces can contain only part of the old session history. **Current behavior** The exporter clips the full model envelope from its start. A sampled trace has a useful HOLD reply but a request that ends inside old history. A truncation flag is not proof that the task failed. **Proposed behavior** Select the task prompt before the text limit. Remove a valid leading generated handoff packet from the excerpt. Link the full retained run event through the existing API. [Source pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/e6d8da04-ca3e-4bd9-bd90-737f7717a9e1). ### Hypothesis and thesis for a fix The baseline exporter spends its text budget on the envelope and old history. The new regression confirms this cause: the received OTLP input loses a short current request after a long handoff. Select the current request in the OpenCode projection before redaction and clipping. Keep storage and access checks with their existing owners. ## What Changed - Select task identity and prompt for known native envelopes. Put pending user messages first in continuation excerpts. - Omit only a valid leading generated handoff packet. Keep plain text and unknown formats on the existing path. - Add a relative run-events reference under the prompt capture switch. Readers must sign in to the source instance and page to the matching turn. - Keep raw tool argument values excluded. Keep redaction, opt-in, capture switches, company access and size limits. - Add exporter, proxy, installed-runtime and private-run retrieval tests. Fix the proxy test's session-directory lookup so it skips the output-custody directory. - Document the excerpt and full-event retrieval path. ## Verification - Baseline `6b6c94a4ea8f3d1b58767d67a7a2fc4ceaecda9c`: the new long-handoff regression failed because OTLP input did not contain the current request. - Same regression passes after the fix. Exporter, redaction and bundled proxy suites: 39 tests passed. - `PAPERCLIP_OPENCODE_QUALIFY=1 pnpm --dir packages/paperclip-runner exec vitest run src/drivers/opencode/company-observability-runtime.test.ts`: 2 passed. Real installed OpenCode 1.18.34 and real HTTP OTLP receiver; local SSE model fixture. Both capture settings pass. The model receives the full original submission. - Private-run API suite: 31 tests passed on base `340157d1d6ec01ad0fa55970e148c6cd332bbcb1` with these changes. Authorized readers retrieve a complete request over 8 KiB through pagination. Other readers receive 404. - Runner TypeScript source and test checks and TypeScript build passed. `git diff --check` passed. - `pnpm verify --explain` returned exit 2 and required broader CI. No full local-suite pass is claimed. - Initial harness attempts used the wrong Vitest working directory, lacked built plugin-sdk, selected the wrong proxy fixture directory, or expected 403 instead of the API's 404. These are not red regression results. The corrected checks above pass. - Review follow-up `dfe995a77f3eb3280a6b081b4df9428fac85f540` removes added code comments required by CONTRIBUTING.md. Exporter + installed OpenCode runtime suites reran: 30 tests passed, with the local SSE model and real HTTP OTLP receiver. `git diff --check` passed. The review thread is resolved. Greptile passed on this exact revision with confidence 5/5 and no actionable findings. Final-head typecheck, build, Runner and E2E checks passed; server test shards and the amd64 Runner image build remain in progress. No hosted model call or deployed recovery is claimed. ## Risks Known envelope excerpts change shape from serialized envelope text to readable task text. The handoff selector depends on the current generated packet format; unknown framing falls back to the full prompt excerpt. Long current requests still clip. Full-event access depends on source-instance permission and retention. Export remains best effort. The new reference grants no access and includes no credential. This PR is unmerged and not deployed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The exact serving variant and context size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Final verification (2026-11-14): exact head `dfe995a77f3eb3280a6b081b4df9428fac85f540` has 60 successful checks, one neutral security scan, and five conditional skips (cache write, publication/attestation, incremental selection, Storybook paths). Applicable typecheck, build, general and serialized tests, Runner tests, E2E and both Runner image architectures passed. [CI](https://github.com/paperclipai/paperclip/actions/runs/43572758306), [Runner images](https://github.com/paperclipai/paperclip/actions/runs/43572758233), [Greptile](https://github.com/paperclipai/paperclip/runs/132380487679). Greptile 5/5, no unresolved actionable findings. Unmerged and not deployed. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1c556d5340 |
fix(native): never rest a task blocked on the agent's own blocker (#22140)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report. A `paperclip_block`
report names a blocker, its owner and the unblock action
> - The status arbiter binds a task-wide blocker. When the owner is not
a person, it binds the blocker to the same agent and wakes that agent
> - The dispatcher refuses those self-wakes
(`chat_control_recovery_proof_unresolved`), because only continuation
and retry effects are proof for a native wake. The task then rests
blocked with no owner, no wake and no Retry
> - #21959 continues a system blocker in a fresh session, but only for
the reason codes `environment_unavailable` and `tool_unavailable`. A
restart-chaos rerun showed the same dead end with the reason code
`tool_output_failure`
> - This pull request makes sure that a blocker that the agent reports
for itself always has a next owner: the automatic environment lane, one
bounded self-continuation, or a person with a Retry that works
> - The benefit is that no task rests blocked with nobody to act, and no
path wakes the agent in a loop
## Linked Issues or Issue Description
Refs #21959
**What happened?**
A native OpenCode run (DeepSeek) said that every tool call returned the
text "cryptic". It reported `paperclip_block` with owner `{kind:
system}`, scope `task_wide`, reason code `tool_output_failure`, and an
unblock action about the tool-output failure. The run succeeded with
disposition `blocked`. The arbiter bound the blocker to the same agent
(`task_wide_blocker_bound`). The `notify-owner` wake coalesced, and the
`blocker-owner` wake was skipped as
`chat_control_recovery_proof_unresolved`. The task rested blocked
(`needs_attention`, 0 blockers), with no Retry and no wake. An earlier
rerun showed the same shape with reason code `environment_unavailable`,
which #21959 fixed for those two codes only.
**Expected behavior**
A blocker that the agent reports for its own tools or environment is a
harness problem. Paperclip continues it automatically, with a bound. A
blocker that the agent says it owns gets one fresh session. A blocker
that needs a person goes to a person, with a clear prompt and a Retry
that starts the task again. Nothing is left with nobody.
**Steps to reproduce**
1. Run a native task. Make the agent call `paperclip_block` with owner
kind `system`, scope `task_wide` and a reason code other than
`environment_unavailable` or `tool_unavailable` (for example
`tool_output_failure`).
2. See the decision `task_wide_blocker_bound` with the agent as the
unblock owner.
3. See both owner wakes refused, and the task blocked with no Retry.
**Paperclip version or commit**
`master` at
|
||
|
|
43b2d8be22 |
feat(dashboard): add an experimental areas and responsibilities dashboard (#22069)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The dashboard is the first page a person sees in a company > - The classic dashboard shows agents, runs, costs and recent tasks > - It does not show what the person is responsible for, or if that work goes well > - People think in areas, such as Fundraising or Customers, and in the things they own in each area > - This pull request adds a dashboard built on areas and responsibilities, behind an experimental setting that is off by default > - The benefit is that a person sees what they own, how it is going, and what agents do for it, on one page ## Linked Issues or Issue Description No public issue exists for this work. The description follows the feature request template. **Subsystem affected** Dashboard, navigation, the task conversation, routines, and the instance experimental settings. **Problem or motivation** The classic dashboard counts agents and tasks. It does not say what a person owns or if it goes well. People who run a company with agents want to see their areas, the responsibilities in each area, the bar each one must meet, and the agents and routines that serve it. **Proposed solution** A person owns responsibilities. Each responsibility belongs to an area and has a bar, a number to reach. Projects, tasks and routines link to a responsibility, and the page shows its status from those links. An agent can interview the person and draft the dashboard. The person accepts or declines each change, and any change can be undone from a saved version. **Alternatives considered** Add the same data to the classic dashboard. That page is a fixed set of cards for company activity, so a second, person-centred page is clearer. The setting lets a person pick one of the two. **Roadmap alignment** This is an experimental feature behind `enableResponsibilities`. It is off by default and changes nothing for a person who leaves it off. ## What Changed - Add the setting "Areas & Responsibilities dashboard" (`enableResponsibilities`, off by default). On, Dashboard in the sidebar opens the new page, and the classic dashboard link is not shown in the sidebars or the command palette. Off, nothing changes. - Add tables and migrations 0492 to 0497 for areas, responsibilities, links to work, metrics, status reports, saved dashboard versions, and model-call records for summaries. - Add the API: areas, responsibilities, links, bars, metrics, status reports, summaries, runs, automation, and the dashboard conversation with versions and undo. Each route checks company access and writes to the activity log. - Add four task conversations with an agent: about a bar, about an automation, about a routine, and about the whole dashboard. The agent proposes a change. Nothing applies until a person accepts it. - Add standing checks: routines that an agent carries out on its heartbeat and reports. - Add the Dashboard page: search, counts, areas that open and close, a primary New area button, and a message box at the bottom of the page. Add the responsibility page with its bar, metrics, work and routines. - Add the Paperclip skill sections and runner API entries that let an agent use these routes. The runner skill inventory is regenerated (166 to 174 rows). - Keep the new runner API tests in their own file so `runner-api.test.ts` stays at 985 lines. - Onboarding is not changed. The onboarding changes that build on this work are a separate pull request. ## Verification - Type checks pass for `packages/db`, `packages/shared`, `server` (all layers) and `ui`. - `pnpm --filter @paperclipai/db check:migrations` passes. A new `drizzle-kit generate` run produces no further change. - Tests for every changed file pass: 26 server test files (3,009 tests), 116 UI test files (1,991 tests) and 28 shared test files (432 tests). - The runner capability inventory checks pass. - I started the app on a local database that already held earlier data. Master's new migrations and these migrations applied without error. With the setting on, the Dashboard and a responsibility page loaded with no console or server errors. The More menu and the command palette show one Dashboard. - I did not run `pnpm verify`, the full local suite, or the browser suites. Required CI covers them. - The agent conversations are tested with simulated agents. No real model has run them. ## Risks - Medium. The change is large, and the setting is off by default. With it off, the new routes refuse requests and the UI is unchanged. - Migrations only add tables and columns. Two replace a CHECK constraint, and one builds an index `CONCURRENTLY`. These two have the `online_compatible` override with a reason in the file, and the second one is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. - Master can add migrations before this merges. The migrations then need new numbers. - `responsibilities.ts` (1,663 lines), `responsibility-summaries.ts` (1,239 lines) and one Storybook fixture file (1,305 lines) are new files over 1,000 lines. They each keep one cohesive unit together: the service for one table family, the model-call path for summaries, and the fixtures that the stories share. No existing file crosses 1,000 lines. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude: Claude Fable 5.2 (`claude-fable-5-2`) wrote most of the change, with Claude Sonnet 5.5 (`claude-sonnet-5-5`) for the final steps. Both used tool use, code execution and browser checks. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f05b90508a |
refactor(core): remove retired compatibility paths (#22055)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core manages chat authorization, AI connection defaults, navigation, and the plugin host. > - Core still writes old default rows and checks an Agent Chat flag that always reads true. > - It also carries a desktop rail that cannot open and private aliases with canonical equivalents. > - This pull request removes those paths and moves their tests onto the paths Core uses. > - The result is fewer Core database operations and less code to maintain. ## Linked Issues or Issue Description **What existing behavior does this improve?** Core AI default writes, Agent Chat execution and delivery, private plugin host and Runner setup, and board navigation. **Subsystem affected** Cross-cutting. Core server, board UI, private Runner helpers, and a repository guard script. **Current behavior** The server writes provider-scoped and per-method AI defaults, then reads only the provider-scoped rows. It reads settings to decide whether to block Agent Chat, although every settings view enables it. The desktop sidebar renders expanded while still tracking requests to open and close a retired icon rail. The agent client retries conflicts that the server already resolves when there is exactly one active match. **Proposed behavior** Write provider-scoped defaults directly. Use the existing Agent Chat authorization and retry checks without the redundant settings reads. Render the current sidebar directly and let the API report true agent reference conflicts. **Reason and benefit** A personal non-custom connection save or explicit default selection sends one fewer SQL statement. It also avoids the provider-default write that the legacy trigger performed when the old statement inserted or updated a row. A successful task-to-chat send or receipt replay avoids two transaction settings SELECTs when the singleton row already exists. Conversation comment, wake release, and retry paths also avoid a transaction settings read when they previously entered the gate. Savings in other settings paths depend on cache hits. Core-to-plugin RPC and plugin-to-Launcher HTTP traffic stay the same. **Breaking changes** The public plugin SDK, host navigation, and organization-switcher fields retain their existing interfaces and behavior. The settings API still accepts and reports the Agent Chat compatibility key. This change needs no schema migration or rollout ordering. **Additional context** Refs #10610 for prior reporting on the retired sidebar rail. Refs #21756 for adjacent Coordinator plugin slots, which remain a separate change. ## What Changed - Stop writing per-method AI defaults. Preserve the provider-scope upsert and first-account behavior. Keep the existing tables and migration trigger. - Remove Agent Chat checks that read a constant true value. Keep conversation ownership, task scope, session, and request-retry checks. - Remove unused lock-owner fields and private plugin lifecycle, secret extraction, Gmail connector, PRP setup, heartbeat resolver, and ACPX aliases. - Use a single API request for agent references. Remove retired sidebar rail state, hooks, and branches across the active and production snapshot layouts. Preserve desktop resizing, mobile drawer controls, contextual navigation, and public plugin slot inputs. - Remove unused FeedCard and AgentDetail inputs and the test-only Git push matcher alias. - Keep the cleanup that erases historical plaintext chat retries from browser storage. Production source in the touched files is 106,800 → 105,633 nonblank lines, down 1,167. The production Git diff adds 251 physical lines and removes 1,480, down 1,229. Those figures exclude tests, fixtures, docs, and Storybook. The entire Git diff adds 482 physical lines and removes 2,325, down 1,843. ## Verification - Merged master at `b56f240f45` into the PR branch as `f73fea4009`. The five sidebar files retain the expanded navigation cleanup and master’s explicit session-replay masking contract. The merge also preserves master’s account-repair recovery guard and GitHub star-prompt behavior. - Focused UI Vitest run passed 361 tests across 11 files. This covers analytics privacy and replay masking, sidebar and layout, GitHub star prompt, and agent detail. - Focused server Vitest run passed 45 tests across three files. This covers restored configuration handling, conversation recovery, and Agent Chat authority and receipt replay. - `pnpm --filter @paperclipai/ui typecheck`, the Git push guard and its 14 tests, and `git diff --check origin/master...HEAD` passed. - `pnpm verify --base origin/master` classified the Git-push guard as a global CI input and returned exit 2. GitHub CI supplied the broader coverage. - GitHub CI and Runner image checks passed on `f73fea4009`. Both required checks, `ci / verify` and `ci / e2e`, passed. [CI workflow](https://github.com/paperclipai/paperclip/actions/runs/44181261752) · [Runner image workflow](https://github.com/paperclipai/paperclip/actions/runs/44181261413) - Standard Greptile reported 5/5 with no actionable findings or open review threads. [Greptile review](https://github.com/paperclipai/paperclip/pull/22055#issuecomment-6959078925) - The optional Superagent Security Scan finished neutral with an inconclusive workflow result and zero annotations. The final rollup had 61 successes, five skips, and one neutral, with no failed or pending checks. [PR checks](https://github.com/paperclipai/paperclip/pull/22055/checks) ## Risks Agent Chat authorization and retry are sensitive to small query changes. The final diff keeps the conversation agent field and all existing authorization and idempotency checks. The tests cover revoked defaults, legacy migration reads, bound-listener authority, and mobile and desktop navigation. Installations that still run software old enough to read the per-method AI table will no longer see new defaults in that table. Current Core reads the provider-scoped table. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with source inspection and shell-based tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
73dc33c620 |
fix(runner): keep a disk reserve so a full sandbox disk does not strand the runner (#22068)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in a Paperclip-managed sandbox run under the Paperclip runner (`runnerd`), which keeps durable state on the sandbox disk > - An agent can fill that disk with its own work. In one case the agent installed a Rust toolchain and a second checkout with dependencies on a 10 GiB sandbox > - `runnerd` then could not write its state and exited. Each relaunch tried to upload its launch files to the full disk and failed, 83 times in 7 minutes > - The run ended as a generic provider failure that asked a person to inspect it, and Retry repeated the same failure > - This pull request keeps a small disk reserve that belongs to Paperclip. Recovery releases it to relaunch the runner, and stops at once with a named code when nothing is left to free > - The benefit is that the agent's turn can continue and see the full disk in its own tool output, instead of the run dying with it ## Linked Issues or Issue Description **What happened?** An agent in a managed Daytona sandbox filled the 10 GiB disk. `runnerd` exited with `failed to persist bounded diagnostic: No space left on device (os error 28)`. Runner process recovery then retried for the full 420000 ms reconnect grace. Every attempt failed with `.paperclip-upload-<id>.tar.gz: create: ... no space left on device`. The run failed as `native_provider_terminal_failed`, and the recovery asked a board user to verify that the provider stopped. **Steps to reproduce** 1. Run an agent in a managed sandbox and have it fill the disk (for example a Rust toolchain plus a full dependency install). 2. `runnerd` exits when it next writes its state. 3. Watch recovery retry the relaunch upload until the grace expires. **Expected behavior** Paperclip relaunches its runner on a disk the agent filled, so the turn continues and the agent sees the full disk in its tools. When the runner truly cannot start, recovery says so at once with a named cause instead of retrying for 7 minutes. **Environment** Managed Daytona sandbox (fleet snapshot, 4 CPU, 8 GiB memory, 10 GiB disk), `paperclip_runner` native runtime. ## What Changed - New `server/src/services/native-runtime/remote-runner-disk-reserve.ts` owns the reserve: hold/release scripts, the full-disk classifier, and `createRemoteRunnerDiskReserve`. A release waits for an in-flight hold, and no hold starts after a release. - Both remote runner launchers (agent runs and Dot) get a reserve path in managed sandboxes only: `<workspace>/.paperclip-runtime/paperclip-runner/disk-reserve`. Workspace saves and Git already exclude `.paperclip-runtime`. SSH hosts get no reserve. - After the first launch's identity is confirmed, the launcher holds 256 MiB in the background, only while 2 GiB more is free. It is off the startup path. - A relaunch releases the reserve before it stages. A launch that hits a full disk while staging releases the reserve and stages once more. - Release tries twice and reports `released`, `not_held` or `failed`. Only `not_held` ends the launch with `sandbox_disk_full`. A release that cannot run rethrows the original error, so recovery still retries. - Command-based staging (no `syncIn`) keeps a full disk recognizable with an `ENOSPC` code instead of a bare `runner_remote_*_staging_failed`. - `SANDBOX_DISK_FULL_FAILURE_CODE` and `isSandboxDiskFullFailure` live in the runner package. Runner process recovery checks the error code and stops at once instead of retrying for the whole grace. Other staging failures still retry. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/remote-runner-disk-reserve.test.ts`: release-and-restage, release before relaunch, `sandbox_disk_full` when release frees too little, no retry for a failure that is not a full disk, hold/release scripts in a real shell, no hold below the free-space floor. Before the fix, the launch rejected with the exact upload error from the incident. - `packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts` "fails at once when a replacement cannot be staged because the sandbox disk is full": before the fix it was still retrying after 4 s; after, it fails with `sandbox_disk_full` after one replacement. - Kept: "retries a replacement whose staging failed before any process launched" still passes, so a raw transient staging failure still retries. - Updated: "confirms no replacement process exists when staging fails before the launch RPC" now expects `sandbox_disk_full` with the upload error as cause. Exit is still confirmed and no launch RPC runs. - Real sandbox from the same fleet snapshot (Daytona, 10 GiB): hold created a 256 MiB reserve. The disk was filled to 100%. A 5 MB write then failed. Release returned `true`, and a 50 MB write then succeeded. A second release returned `false`. Hold on the now 98% full disk did not create a reserve. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. `pnpm verify` result is in a comment below. - Review round: tests for a lost release RPC, hold/release ordering, command-based staging on a full disk, the hold after a dispatched first launch (not on relaunch), and no reserve on SSH. Each launcher-level test failed with its fix removed. - Focused suites on the review revision: transport 335/335, launcher 130/130 (filtered), launch assets 35/35, reserve 14/14. Server and runner typecheck pass. - Not run: a full product run in which an agent fills the disk and the relaunched `runnerd` resumes the turn end to end. ## Risks - Each managed sandbox holds 256 MiB more disk while it has more than 2 GiB free. That is 2.5% of a 10 GiB sandbox. - A relaunch for any reason releases the reserve for the rest of that run. The next run's launch takes it again. - If the agent keeps writing after the relaunch, the disk can fill again. The second time, nothing is left to release, and the run fails at once with `sandbox_disk_full` instead of after 7 minutes. - The run's error code is still `native_provider_terminal_failed`. The message now names `sandbox_disk_full`. A dedicated failure code and recovery text are a follow-up. - `runnerd` loses its own error message when it cannot persist its diagnostic. This PR does not change the Rust binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9911b7052f |
Expose continuation retry keys and preserve account-repair resumes (#22046)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner gives agents a tool to report work and wait for a later check. > - That tool requires a stable retry key for each explicit wait. > - Codex can hide the wait fields when it renders our nested schema. > - Hidden fields make a correct report harder to construct. > - This pull request changes the provider schema structure and preserves its rules. > - Agents can see the required fields before they submit a report. ## Linked Issues or Issue Description **What happened?** The completion tool appears in Codex code mode with `continuation?: unknown`. An agent can omit the required retry key and receive a rejected report. The top-level schema was fixed in #21398. The nested continuation still uses the same problematic structure. **Expected behavior** Codex must show the required `kind`, `summary`, and `idempotencyKey` fields. Invalid reports must still fail validation. **Steps to reproduce** Render `PRP_COMPLETION_RESULT_PROVIDER_INPUT_SCHEMA` with the public Codex code-mode renderer at `b271854568d6c9282e999d6f36e5e85f447d027d`. Its `render_map` function processes `allOf` before sibling properties. The conditional-only branch renders as `unknown` and hides those properties. Source: https://github.com/openai/codex/blob/b271854568d6c9282e999d6f36e5e85f447d027d/codex-rs/code-mode-protocol/src/json_schema_types.rs#L127 **Paperclip version or commit** Baseline: `01451fae5b25f784956e9932f12031dc194ac247`. The first complete CI browser attempt also exposed an intermittent account-repair race. Repair restores the task and resolves its recovery action, then a sweep can reclassify the same old failure and block it again before the queued reply starts. The repair in this PR honors that exact persisted resolution while holding the existing task lock. A later failed run still escalates. ## What Changed - Express the single provider continuation condition as direct `if` and `then` fields. This is equivalent to its previous `allOf` wrapper. - Keep the canonical output schema unchanged. Keep required retry keys and case handoff IDs. - Add a discovery guard, a 60-case provider/canonical validation comparison, and authenticated HTTP bridge checks. ### Hypothesis and thesis for a fix The nested `allOf` hides fields in the tested Codex renderer. The equivalent direct condition lets that renderer reach the object fields. This belongs in the existing provider compatibility schema. It changes no controller policy, company access, saved state, or retry identity. The experiment proves field visibility. It does not prove which renderer served a historical request or measure model omission rates. - Honor a resolved/restored configuration recovery for the same company, task and failed run before re-escalating. Dispatch checks and recovery for new failures remain intact. - Add a PostgreSQL regression covering repeat sweeps, a new failure identity, an unresolved recovery, and another task's repair. This belongs in the existing locked escalation function; no new state or subsystem is introduced. `service.ts` was already over 1,000 lines. - Define the handoff condition once and explicitly compose both canonical and provider schemas from it, removing the implicit one-element `allOf` assumption raised in review. ## Verification - Built and ran the exact pinned public Codex renderer in a disposable Rust crate. The complete baseline schema renders `continuation?: unknown`. The changed schema renders a nullable object with required `kind`, `summary`, and `idempotencyKey` fields. - The new discovery regression fails on the baseline. The same test passes with the fix. - Runner: `pnpm exec vitest run src/contracts/completion-result.test.ts src/contracts/tool-input-variants.test.ts src/contracts/provider-tool-schema-shape.test.ts src/drivers/runner-tool-bridge.schemas.test.ts src/drivers/runner-tool-bridge.continuation.test.ts` — 48 tests pass. - Runner: `pnpm exec tsc -p tsconfig.json` and `node scripts/generate-semantic-contracts.mjs --check` pass. - Server: the selected `task-monitor.integration.test.ts` test for unfinished monitor waits passes with embedded PostgreSQL. It rejects an unscheduled wait and accepts the saved monitor. - Server: the selected `issue-monitor-scheduler.test.ts` recovery test passes with embedded PostgreSQL. It checks persistence, same-key replay, and one scheduled wake. The fixture's successor run fails its separate responsible-user setup. This is not proof of successful successor execution. - HTTP bridge handler is simulated. It verifies rejection before dispatch, valid monitor/response-wake/handoff reports, retry identity, and authentication. - Recovery regression: baseline 1 failed / 3 passed (the restored task was blocked again); unchanged test after the fix 4 passed. Together with recovery state-read tests: 7 passed. Existing configuration-recovery selection: 4 passed, 108 deselected. - Final completion schema and authenticated HTTP bridge selection: 33 passed. - Local real browser journey with disposable embedded PostgreSQL and ACP/provider fixtures: baseline repeat5 produced 2 failures / 3 passes, matching CI. Fixed repeat5: 5 passed in 1.8 minutes. The test independently checks exactly one successful successor, stored connection binding and persisted reply after reload. - No full local suite, deployed recovery, or probabilistic model result is claimed. Fresh CI and review are required for the updated head. - Review follow-up: 48 focused schema/discovery/HTTP tests pass; `node scripts/generate-semantic-contracts.mjs --check` passes. Fresh CI/review required after the refactor. ## Risks Low runtime risk. The accepted input rules are unchanged. A provider that handles direct conditionals differently could expose a compatibility issue. The canonical validator still enforces the output contract. No migration or saved-session reset is needed. Previously saved state keeps the same format. Recovery now honors existing resolved/restored actions only for the exact failed source run, so a new configuration failure remains eligible for escalation. Company, task, ownership and dispatch checks remain in force. ## Model Used OpenAI GPT-6 Codex agent, with reasoning, tool use, and code execution. The runtime did not expose a more specific model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
bc0a236b95 | feat(runner): accept a resume-compatible build for a relaunched released Runner (#21976) | ||
|
|
41b0f76d75 |
fix(runner): tell a provider its accepted result is final when it reports again (#21984)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run reports its result with `paperclip_finish` or
`paperclip_block`, and a turn's first accepted report is final
> - When the model reports again with a different result, the Runner
refuses the second report
> - After a committed blocker, the refusal tells the model to stop and
end the turn (#21959)
> - After a committed done, needs_review or yielded result, the refusal
was only "A different semantic result was already committed", so the
model did not know what to do and tried again
> - Each extra call is one more model request, and the turn holds the
agent's slot until it ends; the task's status commits only when the turn
ends
> - This pull request gives every refused later report a clear, final
message: the result is already accepted, stop calling `paperclip_finish`
and `paperclip_block`, put extra notes in the final response, and end
the turn
> - The benefit is that the model ends the turn after one refusal, so
the slot is free sooner and the task completes sooner
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave, an OpenCode agent (DeepSeek through OpenRouter)
finished a task, and the Runner accepted its `done` report. The model
then called `paperclip_finish` again with a different report. The Runner
replied only "A different semantic result was already committed". The
model repeated the call about 79 times over 277 seconds. The agent's
slot stayed busy, and the task's completion was late, because the task
status commits when the turn ends.
**Expected behavior**
After a turn commits any result (done, needs_review, yielded or
blocked), each later report with a different result gets a clear and
final instruction: the result is already accepted, so do not call
`paperclip_finish` or `paperclip_block` again, put any extra notes in
the final response, and end the turn. An identical repeat of the
accepted report stays an idempotent success.
**Steps to reproduce**
1. Start a native OpenCode turn that calls `paperclip_finish` with a
done result.
2. In the same turn, call `paperclip_finish` again with a different
summary, or call `paperclip_block`.
3. On master, the tool error is only "A different semantic result was
already committed", with no instruction to stop.
**Paperclip version or commit**
master
|
||
|
|
22d3f9a03c |
fix(native): finish a failed warm session's cleanup by itself and continue the task (#21977)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. A follow-up run
can reuse the warm session of the run before it, in a remote sandbox.
> - When a warm follow-up fails, the previous session is closed. The
close saves a backup of the provider session first.
> - The close looked up the backup owner through the task's current tool
authority. The failed follow-up had already released that authority.
Every close attempt failed with `native_harness_storage_owner_missing`,
so the close could never finish.
> - After nine failed attempts, the task's admission gate became
`operator_required`. The gate retires only for a stopped local process
or for the exact run that owns the session. A remote sandbox session
owned by an earlier run never matched. A queued continuation and a Retry
stayed queued forever, and the task showed a Retry button that could not
work.
> - Separately, a remote transport failure could never produce the local
stop proof that the safe-replacement lane waits for. Nothing continued
the task until an unrelated wake arrived 15 minutes later.
> - This pull request makes the close save its backup under its own
owner, retires a cleanup gate when the task's sandbox is confirmed
terminated, and refuses an unprovable remote transport failure at once
so the task continues in a fresh isolated run.
> - The benefit is that recovery finishes by itself, and a Retry that is
shown starts a run.
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave, a warm follow-up run on a remote sandbox failed
with `provider_transport_failed` (`PRP command run.attach timed out`).
The previous run's session close then failed on every attempt with
`native_harness_storage_owner_missing`. The cleanup gate became
`operator_required`. The continuation run was created 15 minutes later
and then stayed queued until the wave ended ("queued run kept queued
while the task's previous session finishes cleanup"). The task showed a
Retry banner, but a Retry would wait behind the same gate.
**Expected behavior**
A failed session close settles or is retired by itself with evidence.
The next run of the task starts in seconds. A Retry that is shown starts
a run.
**Steps to reproduce**
1. Run a native task on a remote sandbox with a warm session lifecycle.
2. Make the warm follow-up run fail during startup, after it has reused
the previous session.
3. Watch the previous session's close: every attempt fails with
`native_harness_storage_owner_missing`, and the queued next run of the
task never starts.
**Paperclip version or commit**
master at
|
||
|
|
7685b7a4fc |
fix(native): continue a run that reports a system environment blocker (#21959)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report: `paperclip_finish` or
`paperclip_block`
> - An agent can report a system environment blocker, for example when
it thinks its tool results are corrupted
> - Today that blocker binds to the same agent, its wake is dropped, and
the task stays blocked with no Retry
> - In a restart-chaos rerun, an OpenCode agent reported such a blocker,
then recovered in the same turn and did all the work, but its later
`paperclip_finish` was refused with an opaque conflict error
> - This pull request makes Paperclip continue a system environment
blocker automatically in a fresh session, and tells the agent clearly
that a turn's accepted blocker is final
> - The benefit is that a temporary environment problem no longer needs
a person to unstick the task
## Linked Issues or Issue Description
**What happened?**
A native OpenCode run reported `paperclip_block` with owner `{kind:
system, name: execution-environment}`, `reasonCode:
environment_unavailable` and the unblock action "Resume/retry the run".
In the same turn the agent recovered: it saved the task document and
posted the requested comment. Its later `paperclip_finish` failed with
"A different semantic result was already committed". The run succeeded
with disposition `blocked`. The status decision bound a task-wide
blocker to the same agent. Both owner wakes were skipped
(`chat_control_recovery_proof_unresolved`). The task stayed blocked with
no Retry.
**Expected behavior**
A system environment or tool blocker is a harness problem. Paperclip
must continue the task automatically, with a bound, in a fresh session.
A provider that tries to replace an accepted blocker must get clear
guidance, not an opaque conflict.
**Steps to reproduce**
1. Run a native task with an agent that calls `paperclip_block` with
owner kind `system` and reason code `environment_unavailable`.
2. In the same turn, call `paperclip_finish` with a done report.
3. See that the finish is refused with "A different semantic result was
already committed" and that the task stays blocked with no Retry.
**Paperclip version or commit**
`master` at
|
||
|
|
fb288f5666 |
fix(acpx): only treat an error result as a Claude sign-in failure (#21921)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the Claude ACP bridge (`claude-agent-acp` 0.73.0), which Paperclip patches in `patches/` > - The bridge reads Claude Code's final result for each turn. If the result text contains "Please run /login", the bridge fails the turn as a sign-in failure > - That check does not look at `is_error`. A successful reply that quotes the CLI's sign-in hint also fails the turn > - Agents quote that hint when they report the output of a `claude` command that they ran. The work is done, but the run fails with "ACP agent reported a terminal access failure." > - Upstream fixed this in [agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103) (commit [`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f), released in 0.81.1), "fix: gate login result text on error state" > - This pull request ports that upstream change into the 0.73.0 bridge patch > - The benefit is that a correct, finished run is not reported as a failed sign-in ## Linked Issues or Issue Description Upstream fix: [agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103) (commit [`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f), released in 0.81.1). This pull request ports it. Refs #14089. That pull request fixes the same class of problem in the Claude CLI lane (`execute.ts`). This pull request fixes the ACP lane. Related: #14093 shows the same error text, but it has a different cause. This pull request does not fix #14093. **What happened?** An agent ran a script that calls the `claude` CLI without credentials. The CLI printed "Not logged in · Please run /login". The agent finished its task and quoted that text in its final reply. Claude Code ended the turn with a successful result (`is_error: false`). The bridge saw "Please run /login" in the result text and failed the turn with `auth_required`. Paperclip then failed the run with "ACP agent reported a terminal access failure." **Expected behavior** A successful result completes the turn. Only an error result with the sign-in hint is a sign-in failure. **Steps to reproduce** 1. Run a Claude agent on the ACP lane. 2. Make the model's final reply contain the text "Please run /login" (for example, quote the output of `claude -p hi` without credentials). 3. See that the run fails with "ACP agent reported a terminal access failure." The new test in this pull request does these steps against a local API stand-in. **Agent adapter(s) involved** `claude_local` on the ACP engine, and the Runner's ACPX Claude profile. Both use the same patched bridge. ## What Changed - `patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch`: port [agentclientprotocol/claude-agent-acp#1103](https://github.com/agentclientprotocol/claude-agent-acp/pull/1103) (commit [`6e97e07`](https://github.com/agentclientprotocol/claude-agent-acp/commit/6e97e071c581002b43d9556dbd1d1376ebf9516f), released in 0.81.1). It has two parts, the same as upstream: - The `success` result check is now `message.is_error && message.result.includes("Please run /login")`. A successful result that quotes the sign-in hint no longer fails the turn. - The goal-command check also requires `message.is_error`. A `/goal` command whose successful result quotes the sign-in hint no longer rolls the session goal back. - Both patch comments name the upstream change. All other patch hunks are unchanged. - `packages/paperclip-runner/src/drivers/acpx/acpx-runtime-adapter.claude-login-text.test.ts`: a new test. It runs the real bridge and the bundled Claude Code against a local API stand-in. The model's reply quotes the sign-in hint, and the test expects the turn to complete. ## Verification - Red: on `master`'s patch, the new test fails with `{"status":"failed","error":{"message":"ACP agent reported a terminal access failure."}}`. - Green: with this patch, the new test passes (`status: "completed"`, `stopReason: "end_turn"`). - I ran Claude Code 2.1.286 (the bundled version) with no credentials. Its real sign-in failure is a result with `is_error: true` and the text "Not logged in · Please run /login". The guard still catches it. The separate synthetic sign-in message check is unchanged. - The goal-command part has no new test here. The upstream change adds a test for it. The runner's `session-goals.test.ts` passes with the new patch. - `node --test scripts/acpx-patch-packaging.test.mjs`: 26 passed. - `node --test packages/paperclip-runner/test/acpx-codex-package-contract.test.mjs`: 9 passed. - Runner real-bridge Claude tests (`acpx-runtime-adapter.claude-*.test.ts`, `acpx-runtime-adapter.test.ts`, `claude-scheduling-tools.claude-acp.test.ts`, `runner-tool-bridge.claude-*.test.ts`): 139 passed. - `claude-local` ACP tests (`acp.test.ts`, `acp.auth.test.ts`, `acp.quota.test.ts`): 95 passed. - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`: passed. - I did not commit `pnpm-lock.yaml`. The patch hash changes, and CI resolves the lockfile for this run. ## Risks - Low risk. The change adds one condition to each of two checks in the bridge patch. Both match the upstream change. - The bridge's `commandDigest` does not change, because the patch does not touch `dist/index.js`. Saved sessions are not reset. - A sign-in failure that Claude Code reports as a successful, non-error result is no longer caught by this check. Claude Code 2.1.286 reports sign-in failures as error results, and upstream uses the same rule. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, subagents) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0308108531 |
fix(native): pipeline tool progress events with text deltas (#21533)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs store every relayed runner event in `heartbeat_run_events`. Each stored event also locks and updates the run's `heartbeat_runs` row. > - Text deltas already commit in batches. Every other event, including each `tool.execution.progressed` event, commits in its own transaction. > - Tool receipts and usage accounting lock the same run row. Under parallel tool calls, accounting failed with "other updates to the same run kept conflicting", and a provider-pack receipt save hit a lock timeout. > - #21520 made events that arrive while a batch commits join the next batch. This pull request adds tool progress events to the pipelined types, so they use those batches too. > - The benefit is fewer run-row transactions in tool-heavy turns, without changing event order. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents that made about 190 tool calls in about 10 minutes. Three tool calls failed with "could not save this run's budget or usage records because other updates to the same run kept conflicting", and a provider-pack admission receipt failed with a lock timeout (55P03). Every tool progress event of the run was a separate transaction that updated the same run row. **Expected behavior** Observational events such as tool progress commit in batches, so they hold the run row lock fewer times. **Steps to reproduce** 1. Run a turn that makes many tool calls in parallel. 2. Each `tool.execution.progressed` event commits in its own transaction and updates `heartbeat_runs`. **Relevant logs or output** ``` could not save this run's budget or usage records because other updates to the same run kept conflicting native provider-pack admission receipt could not be saved (cause=55P03) ``` Related: #21531 tried letting the runner relay events faster, which would make this contention matter more. It is closed. ## What Changed - `paperclip-control-plane-port.ts`: - `pipelinedEventTypes` names `item.delta` and `tool.execution.progressed`. Both go through the batched append path and its group commit from #21520. - The text-delta boundary is now tracked when an event is appended: a pipelined event that is not a delta ends it, and a delta with text starts it. The first text delta after any other event still commits at once. - `contracts/control-plane-port.ts`: optional `pipelinedEventTypes`, and the `streaming` option documents that it applies to those types. - `native-session-runtime.ts`: the runtime pipelines the port's `pipelinedEventTypes`, and only `item.delta` when the port does not name them. - Tests: - `paperclip-control-plane-port.test.ts`: three progress events appended while a batch commits share one transaction, and a following event commits after them. - `native-session-runtime.test.ts`: progress appends start without waiting only when the port pipelines them. ## Verification - After rebasing onto current master: `pnpm exec vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts` (in `server/`): 108 passed. - `npx vitest run src/native-session-runtime.test.ts` (in `packages/paperclip-runner`): 298 passed, 1 failed. `settles a governed wait through real Runner shutdown after a missing Codex terminal` needs the built Rust test binaries (`fake-codex-app-server`), which my checkout does not have. CI builds them. - `pnpm exec tsc --noEmit -p .` in `server/` (after building its workspace dependencies) and in `packages/paperclip-runner` pass. - `npx vitest run src/__tests__/chat-channels.integration.test.ts src/__tests__/heartbeat-process-recovery.test.ts src/services/native-runtime/native-plan-wait.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts` (in `server/`): 2,586 passed, 2 failed. `binds Slack tools to the admitted linked request...` reads a skills checkout that is not next to my worktree. `materializes direct external-chat finals once...` failed in the full file run but passes alone, with and without this change. ## Risks - Progress events now commit up to one transaction later than before. Their receipts and live UI updates follow their batch. - A batch that contains progress events still writes rows one by one inside its transaction, because progress events register chat-progress wakes. The gain is one transaction and one lock wait per batch, not one row update per batch. - If a batch fails, every event in it fails, and the runner resends them. That is the same as for text deltas today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
46e5e4450f |
fix(runner): leave an undispatched tool call to the next controller on restart detach (#21786)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner keeps an agent session alive while its controller restarts, for example during a deploy, and the next controller adopts it > - A Paperclip tool call can be between its committed input and its handler when the controller detaches > - In that window the old controller journaled "The turn stopped before this operation was dispatched" as the final, non-retryable result, but the turn did not stop > - The agent then got a false failure for a call that never ran, and the next controller could not recover the call. A restart must never fail work > - This pull request leaves that call unsettled for the next controller, the same as a call that was already in its handler > - The benefit is that a controller restart no longer turns an agent's pending tool call into a permanent error, and a CI test that failed on four pull requests now passes deterministically ## Linked Issues or Issue Description **What happened?** `src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` › `settles an earlier generation's in-flight tool call at attach even when that attach never finishes startup` failed in "ci / Verify Paperclip Runner (vitest package (3/3))" on 2026-11-15. In each job the first try and the CI retry both failed: - PR #21757, head `d04825d812`, job 132824209377 - PR #21713, head `7758d56719`, job 132681892394 - PR #21729, head `c63a05a013`, job 132732839971 Every failure is the same, at line 74, right after `detachControllerForRestart()`: ``` AssertionError: expected [ { …(8) } ] to deeply equal [] + "type": "semantic_tool.result", "status": "pending", + "result": { "error": { "code": "semantic_tool_not_dispatched", + "message": "The turn stopped before this operation was dispatched.", "retryable": false } } ``` **Expected behavior** When a controller detaches for restart, a tool call that it has not dispatched stays unsettled. The next controller recovers it at attach, the same as a call that is in its handler. **Steps to reproduce** 1. Hold the semantic tool dispatch at a gate after the Runner commits the tool input. 2. Call `detachControllerForRestart()`, then release the gate. 3. On `master`, the control-plane state has the `semantic_tool_not_dispatched` result above. It is byte-identical to CI (same command id and payload). The new test case `(still before dispatch at detach)` does this. **Root cause** This is a Runner defect, not a test timing problem. `#detachControllerOnce` sets `#closed`, resolves the closed signal and resolves the turn-start admission with `false`. A call in `#handleSemanticToolInput` that has not reached the handler then throws `SemanticToolNotDispatchedError`. The durable control plane turns that error into a journaled `semantic_tool_not_dispatched` result. That result is correct when the turn stops. It is false when the controller only detaches for restart, because the Runner and the turn continue. The old test waited only for the committed input, so on a loaded CI runner the detach sometimes came before the handler call. **Production impact** Yes. During a controller restart or deploy, an agent tool call in this window got a non-retryable "turn stopped" error after the Runner was adopted, and the next controller did not recover the call. ## What Changed - `durable-prp-control-plane.ts`: new `SemanticToolHandedOffError`. When the dispatcher rejects with it, the control plane journals no result. The input stays unsettled for the next controller. - `runnerd-codex-transport.ts`: the semantic tool callback goes through `#dispatchSemanticToolInput`. If the call ends as not dispatched after the transport detached for restart, it throws `SemanticToolHandedOffError`. A normal close still journals `semantic_tool_not_dispatched`. - `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: the test waits for the exact stage instead of the committed input. It runs two cases: the call is in its handler at detach, and the call is held before dispatch across the detach. Both then check the existing attach recovery and the final `completed` result. ## Verification - Red on `master`: the `(still before dispatch at detach)` case fails every run with the CI assertion and payload. - Green on this branch: `npx vitest run src/live/runnerd-codex-transport-orphaned-tool-at-attach.test.ts` passes 2/2. - Loop on macOS with 4 extra busy-loop processes (load average about 13 before them), 50 runs each: this branch 50/50 (100 test cases). The old `master` test was 50/50 locally too, so the local loop does not show the CI interleaving. The gate does. - All 36 runner test files that use `detachControllerForRestart` or `SemanticToolNotDispatchedError`: 1180/1182 pass with 3 workers under load. The 2 failures (`runnerd-codex-transport-settlement.test.ts` and `runnerd-acpx-source-reconciliation.test.ts`, `kill EPERM`) pass 36/36 when run alone. - `npx tsc -p tsconfig.json --noEmit` and `npx tsc -p tsconfig.test.json --noEmit` pass in `packages/paperclip-runner`. - The other two static-check flakes from the same day are test-only and are fixed in a separate pull request. ## Risks - Medium-low risk. This changes Runner restart behavior for one window. A call that was not dispatched now goes to the next controller's recovery (`recoverSemanticToolInput`) instead of getting a final error. That is the same path that already handles a call in its handler, so no new recovery code is involved. - If no successor ever attaches, the input stays unsettled. That is the same state as for a call that was in its handler at detach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
348ed0ae10 |
fix(runner): keep closing tool-call tags and model control tokens out of OpenCode chat (#21761)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner OpenCode driver streams model text into the
chat as progress and selects the final reply
> - DeepSeek v4.1 flash (through OpenRouter) wrote tool-call scaffolding
and tokenizer control tokens as ordinary text
> - #21361 hides tool-call markup, but its detector matched only opening
DSML tags, and nothing recognised control tokens
> - A person saw raw `</|DSML| invoke>` lines in a progress message, and
in another conversation got a degenerate reply instead of an answer
> - This pull request extends the same detector to closing DSML tags and
to model control tokens, on the streaming path and at completion
> - The benefit is that scaffolding never reaches the chat, and a
degenerate reply goes to the existing correction instead of being posted
## Linked Issues or Issue Description
Refs #21361
**What happened?**
Chat-eval cohort 16 (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
|
||
|
|
01c1e708e9 |
fix(runner): stop a turn that keeps resending a refused tool call and continue fresh (#21734)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs call Paperclip tools, and
`RepeatedToolFailureGuard` refuses an exact repeat of a call that
already failed validation twice
> - The refusals are not counted and never end the turn, so a model
whose host drops an argument can resend the same refused call until the
run's wall clock
> - In a recorded restart-chaos wave, an opencode/DeepSeek agent sent
the same `PATCH /api/issues/{id}` through `call_api` 431 times in 40
minutes without `body`; 427 were refused. The run held an agent slot and
delayed other tasks
> - The system knows the next step (stop the turn and continue on a
fresh session), so the harness must own it, with no Retry banner
> - This pull request counts refusals, stops the provider turn after a
bounded number with the classified cause `agent_tool_call_loop`, and
queues an automatic fresh-session continuation that tells the model to
use a dedicated tool
> - It also records the declared argument names that each opencode tool
call delivered, so we can tell whether the bridge or the model drops
`body`
> - The benefit is that a looping agent frees its slot in seconds
instead of 40 minutes, and the task continues by itself
## Linked Issues or Issue Description
**What happened?**
An opencode agent on DeepSeek created two subtasks and then tried to set
their descriptions with `call_api` `PATCH /api/issues/{id}`. Every call
arrived with `operationId`, `pathParams`, `query` and `responseText`,
but without `body`. The guard refused each repeat. The model said "I
need to include body" more than 20 times and kept sending the same call:
431 calls, 427 refused, over 40 minutes. The run stayed active and its
agent slot delayed other queued work.
**Expected behavior**
After a small, bounded number of refusals, Paperclip stops the provider
turn and continues the task automatically on a fresh session, with a
hint to use a dedicated Paperclip tool. The board shows no Retry banner
because the system owns this recovery.
**Steps to reproduce**
1. Run a native agent whose model resends one tool call with a missing
required argument.
2. The guard refuses the third and later identical calls.
3. On master, the refusals never end the turn.
**Paperclip version or commit**
`master` at
|
||
|
|
a78be8b601 |
feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment plugins can run a native Runner as a plugin task, and a warm session keeps that Runner between turns > - On each warm turn Core sends the plugin a `prepare` operation with the full launch requirements for the new run > - The requirements' `requirementsDigest` covers the run identity, and the run identity changes on every turn > - So a plugin cannot tell that only the run moved, and it must redo its whole preparation every warm turn > - This pull request gives the plugin two digests: one that stays the same while its preparation stays valid, and one that names the run > - The benefit is that a plugin can transfer only the run identity when nothing it reconciles changed ## Linked Issues or Issue Description **What existing behavior does this improve?** The `prepare` operation that Core sends to an environment plugin for a plugin Runner task, on cold starts and on warm attaches. **Subsystem affected** Native runtime, plugin Runner tasks, the Runner launch contract, and the plugin SDK environment task contract. **Current behavior** A warm attach sends the full requirements. The only digest available, `requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it changes on every turn even when the launch is otherwise the same. **Proposed behavior** Core computes two digests for every `prepare`: - `preparationDigest`: the launch requirements an environment reconciles when it prepares a Runner: - the runner, lease and session identity; - provider, model and settings; - the prompt and instruction text; - the runtime context's instruction and skill bundles, working copy and connection instructions. It leaves out the `runId`, `turnId` and `itemId`, and the runtime context's MCP assignment and binding (with `aggregateDigest`, which hashes the assignment). MCP reaches the Runner through Core's tool authority on every run attachment, never through files the environment publishes. Core already continues a warm session across an MCP-only change (`nativeRuntimeContextsMatch` with tool refresh). The binding is also named per run (`native-mcp:<runId>`). So the digest stays the same across warm turns of one Runner, lease and session, including when the agent's MCP access changes. - `runDigest`: `preparationDigest` plus everything it leaves out: `runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is unique to one run's launch. A plugin that advertises `launchDigestVersion: 1` in its task description gets both fields on `prepare`. **Reason and benefit** A plugin can compare `preparationDigest` with its saved preparation and transfer only the run identity when they match, instead of a full preparation. **Breaking changes** None. Plugins without `launchDigestVersion` get no new fields. ## What Changed - `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add `runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The preparation digest drops the MCP assignment and binding; the run digest takes the full requirements and includes them. Both use the same canonical JSON as `runnerLaunchRequirementsDigest`, with their own schema tags so the three digests never collide. Export them from the package and the server's vendored runner shim. - `packages/plugins/sdk/src/environment-tasks.ts`: add optional `preparationDigest` and `runDigest` to the `prepare` operation, and optional `launchDigestVersion: 1` to the task description. The capability gates both digests. - `server/src/services/native-runtime/plugin-runner-task.ts` and `native-session-executor.ts`: send both digests on `prepare` when the description advertises `launchDigestVersion: 1`. - `server/src/services/environment-task-runtime.ts`: recompute both digests from the requirements, and refuse a `prepare` whose digests are missing in part or do not match. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK public source pin. - Tests: - the preparation digest is stable across runs and changes when the runner, lease, session, instructions, model, approval policy or settings change; - with a runtime context, an MCP-only change (the run's binding, no binding, a changed access set) keeps the preparation digest and changes the run digest; - a change to the instruction bundle, instruction entry, a skill bundle, the skill set, the working copy or the connection instructions changes the preparation digest; - the run digest changes per run; - Core sends the digests only when advertised; - the runtime refuses forged or partial digests before it calls the plugin. ## Verification - `npx vitest run src/live/runner-launch-preparation.test.ts` in `packages/paperclip-runner`: 26 passed. - `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed. - `npx vitest run source-sdk-public-inputs.test.mjs source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in `tests/runner-e2e`: 59 passed. - `pnpm typecheck` in `server` and in `packages/paperclip-runner`: clean. - Not run locally: the full suite. CI covers it. ## Risks Low risk. The digests are new optional fields behind a plugin capability version. Core computes them and checks them again before it calls the plugin, so a plugin can rely on them. They prove only that the launch requirements are unchanged. A plugin that skips work on a match must still check its own state, such as files it published into the sandbox. If the runtime context carries values that change per run, the preparation digest changes too, so a plugin does its full preparation. That is safe, but the plugin saves nothing. The MCP binding is the one such value Core sets per run today, and MCP is excluded entirely. A plugin that does publish MCP configuration into its sandbox must not rely on this digest for it. Values outside the requirements are not covered at all: the account identity and credentials, the GitHub broker, project IDs, and the plugin's own runtime revision. A plugin must keep checking those itself. A match lets a plugin skip publishing its launch configuration, not computing the descriptor: `prepare` must still return a launch whose `requirementsDigest` matches the current requirements, which carry the current MCP assignment and binding. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c603f591dc |
fix(runner): bound ACPX instructions only where the sandbox delivers them (#21646)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner prepares a private sandbox for each ACPX agent before it
opens the provider session
> - `validateProviderPolicy` in that sandbox stage refuses system
instructions over 32 KiB for every ACPX agent
> - Only Pi gets the text from the sandbox stage. Claude, Codex, Grok
and Antigravity do not use it there. The host accepts 256 KiB
> - An agent with a large instruction bundle fails every run at
`session.open`. The error says only `classification=admission_sandbox`,
so the operator cannot see the cause
> - This pull request applies the 32 KiB sandbox bound only to Pi, and
gives every instructions rejection its own closed admission code
> - The benefit is that Claude and Codex agents with large instructions
can start, and an over-bound rejection names its real cause
## Linked Issues or Issue Description
Fixes: #20814
Refs #14430 (this pull request added the 32 KiB check, with no stated
reason for the other providers)
Related: #21644 (also fixes #20814, but only for Claude, and does not
change the classification)
**What happened?**
Every run of a Claude ACPX agent failed with
`provider_session_open_rejected`. The message was `ACPX sidecar command
session.open was rejected (retryable=false,
classification=admission_sandbox)`. A reproduction in a sandbox from the
same image showed that all the filesystem steps of
`prepareAcpxRuntimeSandbox` succeed. The real cause: the agent's
instructions entry file was 54,734 bytes. That is more than the 32 KiB
check in `validateProviderPolicy`. The host bound (`boundedInstructions`
in `runtime-host.ts`) is 256 KiB. Agents with smaller instructions (12.7
KB) work.
**Expected behavior**
The sandbox stage must refuse only the text that it delivers. #14430
added the check when only Pi consumed `systemInstructions` in the
sandbox (as the `PAPERCLIP_PI_SYSTEM_INSTRUCTIONS` env var). The check
runs before any per-provider branch, so it also refuses Claude, Codex,
Grok and Antigravity. Those providers get the text in the session
request. When a bound refuses the text, the error must name the
instructions, not the sandbox.
**Steps to reproduce**
1. Create a Claude ACPX agent whose composed instructions are more than
32 KiB (for example, an `AGENTS.md` of 54 KB).
2. Start a run of the agent on the Paperclip Runner.
3. See that `session.open` fails with
`classification=admission_sandbox`.
**Paperclip version/commit**
`master` at
|
||
|
|
ddfbbfff20 |
fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. #21607 lets a caller ask the transport for it. > - A controller can restart at any time, for example at a deploy. A wait can last 10 minutes and a provider-started turn can last longer, so a restart during one of them is a normal event. > - Tests with the real `paperclip-runnerd` showed that such a restart failed the session. The checkpoint that Core had was the checkpoint of the turn before, and the recovery of the driver refused or mislabelled the turns that it did not know. > - This pull request makes that restart safe. It changes no production run, because no caller asks for provider-started turns before part 3. > - The benefit is that part 3 can turn the behavior on without a failed run at each deploy. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. **This pull request:** the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. This is part 2. Its base branch is `master`. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It makes a controller restart safe for a run that asked for provider-started turns. **Subsystem affected** Paperclip Runner package: the consumer (`src/native-session-runtime.ts`), the session backend (`src/backends/harness-driver-backend.ts`), the driver (`src/drivers/codex/`) and the transport (`src/live/`). Only for a run that asked for provider-started turns. **Current behavior** - The consumer saves no checkpoint when a wait starts or when a provider-started turn starts. During a wait and during a provider-started turn, the checkpoint that Core has is the checkpoint from the start of the turn before. It names that turn as active. - A restart from that checkpoint fails: the driver has the old turn as active, gets the events of a provider-started turn, and stops the session with `turn_binding_mismatch`. - The recovery of the driver adopts one turn that it does not know as a disposition recovery turn, and refuses two or more. With two or more, it closes the transport, and that stops the Runner and Claude. - The Runner does not send again an event that the controller acknowledged. If the start of a provider-started turn was acknowledged before the restart, the new driver gets the other events of that turn with no start. - A turn that ends waiting for a provider-started turn sets the terminal of the session, as a turn that ends the run does. After a restart, if the provider-started turn already has its result when recovery completes, the consumer finds a result and a terminal, settles the run, and does not read the events of that turn. The start, the items, the usage event and the terminal of that turn do not reach Core. - The recovered driver reads notifications from its construction, and `reconcile()` maps the terminal of the saved turn only after its reads. If the saved checkpoint is still the checkpoint of the controller turn and the start of the provider-started turn is already queued, the driver judges that start too early and stops the session with `turn_binding_mismatch`. - A checkpoint that waits can also have a consumed disposition marker (a recovery turn after an earlier restart ended waiting). The driver restores that session as closed and reads no notifications, and the consumer settles the run on the earlier terminal with no result. **Proposed behavior** - **Checkpoints.** The consumer saves a checkpoint when a wait starts and when a provider-started turn starts, with the function that it already uses for a governed wait. A checkpoint that fails stops the turn, as for a governed wait. - **A checkpoint as of the event that Core stored.** The driver gives each event its sequence when it reads it from the provider, before the consumer stores it. A snapshot that is taken later can be ahead of Core. For these two checkpoints, the driver captures its snapshot when it has handled the notification that made the event (the `turn.completed` that waits, or the `turn.started` of the provider), with the sequence of that event. The consumer asks for the snapshot as of the event that it has just appended. The saved state and cursor are then those of an event that Core has. - **Session terminal.** A turn that ends waiting for a provider-started turn does not set the terminal of the session (`HarnessNativeSession`). The terminal of the session is the end of the run, and that turn does not end the run. A checkpoint that is saved during a wait or during a provider-started turn has no terminal, so a recovered session settles only from the events that it reads, in order, to the terminal of the turn that has the result. - **Driver, a checkpoint that waits.** Recovery keeps the wait. It does not adopt and does not refuse the turns after the last terminal that it knows. The Runner sends again, in order, each event that was not acknowledged, and the driver admits each provider-started turn from that stream, as it does with no restart. - **Driver, a checkpoint with an active turn.** `reconcile()` accepts one turn in progress that comes after the saved active turn, when the saved turn ended with the wait marker. It refuses each other unexpected active turn as before, and it refuses this one too when the reconciled turn settled a result. - **Driver, a start that is queued before recovery reconciled the saved turn.** A `turn/started` of the provider that arrives while the session does not wait for one first waits until `reconcile()` has settled, and is then judged as before. `reconcile()` judges the live state after its reads; if the stream already settled the saved turn into a wait, it accepts exactly one turn in progress after the last terminal turn and does not adopt it. The driver admits that turn at its start. - **A wait with a consumed disposition marker.** The driver keeps reading notifications for a restored session that waits, as it does for a wait with no restart. It still refuses a second controller-started turn. The consumer does not take the consumed-marker shortcut for a recovered snapshot that waits, so the wait is resumed. - **Transport, the start comes first.** For each driver, the transport gives the `turn/started` of a provider-started turn before each other notification and before a tool call of that turn. If it did not give that start to this driver, it builds it from the `turn.started` event in its stored events. It does not do this for the turn that the recovered driver already has as active (`resumeActiveTurnId`). It binds the events and the tool call of a provider-started turn to that turn, not to the turn that the Runner reports as active at the time of the recovery read. - **Transport, which start owns an event.** The stored events can have gaps: the window removes old events but keeps the latest of each anchor type, and the controller checkpoint drops streamed deltas. The owner of an event is the last stored `turn.started` before it, if it is a provider start, and if the first turn terminal that is stored after that start is not before the event and is the terminal of the same provider turn (`providerTurnId`). If the first stored terminal after the start is the terminal of a different turn, the end of the provider turn is not known, and the transport builds no start. **Reason and benefit** - The checkpoint is the cause. With a true checkpoint, the two states of a restart are states that recovery can handle: a wait, or an active turn. - The Runner already sends again each event that was not acknowledged. Recovery uses that stream and does not build the turns again from a read. - The window removes a terminal only while a later terminal of the same type is stored. Thus the first stored terminal after a start shows if a terminal of that turn was removed. A rule that only ignores the gaps is not safe: the Runner can make a `turn.failed` for a turn that did not emit its start (`provider_turn_lost_on_restore`). ## What Changed Production code: 113 lines added, 19 removed. - `src/native-session-runtime.ts` (15 added, 4 removed): the two checkpoints in `consumeTurn`, with the sequence of the appended event, and the resumed wait with a consumed disposition marker. - `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed), `codex-session-terminal.ts` (2 added), `codex-session-notifications.ts` (2 added), `codex-harness-session.ts` (24 added, 7 removed), `codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at the event, and the recovery of the driver. - `src/contracts/harness-driver.ts` and `src/contracts/native-session-backend.ts` (1 added each): an optional `boundarySnapshot` on a driver session, and an optional sequence in the snapshot options. Other drivers do not implement it and are not changed. - `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the session terminal, and the snapshot as of a sequence. - `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and `src/live/provider-turn-tool-order.ts` (18 added): the start-first rule and the owner of an event. - Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5 removed), `src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348 added), `src/live/runnerd-codex-transport.test.ts` (137 added), `src/control-plane/durable-prp-control-plane.test.ts` (84 added), `src/live/provider-turn-tool-order.test.ts` (new, 61), `src/native-session-runtime.provider-turn.test.ts` (92 added, 6 removed), `src/backends/harness-driver-backend.test.ts` (56 added). - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only, 123 added, 14 removed): a delay before a provider-started turn, a goal state at its terminal, a list of provider-started turns for one session, a gap between them, a turn that stays open until the new controller reads the snapshot, and a turn that starts after that read. The modes that existed are not changed, and the fake reads its input on one thread as before. ## State saved by the previous release - No new persisted field. The driver session already has `awaitingProviderTurn` (#21453). - **More checkpoints:** one when a wait starts and one when a provider-started turn starts. An ordinary turn saves the same number of checkpoints as before (4 in the unit test, measured before and after). - **No terminal in a checkpoint of a wait.** A checkpoint that is saved during a wait or during a provider-started turn has `terminal: null`. When a wait ends with no provider-started turn (the limit, a comment of a person, or the Runner ends the run), the last checkpoint of the run also has `terminal: null`; before, it had the terminal of the turn that waited. One reader uses that field with no result: the text of a cleanup failure (`nativeCheckpointRecordsCompletedTurn`), which then does not say that the turn completed. In the tests with the real daemon (the limit, the provider process lost, the sidecar exits), the run still ends on the path of today (`native_finalization_missing`). - **A checkpoint of the previous release** has no waiting state. Recovery takes the path of today. - **Rollback of the Runner package during a wait:** the previous code reads the checkpoint, ignores `awaitingProviderTurn`, and takes its recovery path of today. ## Verification With nothing from part 3. The Rust checks and the build checks in the first item were run before the last seven commits; these commits change no Rust. The other items were run at the head of this branch (`36d5e5e587`), or at an earlier commit of this branch for a suite whose files the later commits do not change. - Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm run build:typescript` and `node --test test/acpx-sidecar-contract.test.mjs` (18 passed) in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt --check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the `acpx` unit tests (102 passed): all passed. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in `server`: all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the production code of this head, 26 of 26 passed in each. The Core of these tests follows the production append (`appendNativeSourceEvent`): it stores an event that is out of order, keeps its contiguous cursor below a gap, and reports a duplicate or a conflict for an event that it has. It does not reject a gap in the sequence; production does not. The restart cases, all with the real `paperclip-runnerd`: - **From the checkpoint that the real consumer saved.** The first `executeNativeSession` is fenced and detached as the server does for a restart, and a second one recovers from the last checkpoint that the first one saved. (a) Restart during the wait; the provider-started turn starts after the new controller is attached. (b) Restart while a provider-started turn is in progress. In each: the test asserts the shape of that checkpoint, the run settles with the result of the provider-started turn, each event is appended one time and in order, the turn has one `turn.started` with `origin: "provider"`, and the session closes with no error. - **Without the two checkpoints, each of these cases fails:** the saved checkpoint is the checkpoint of the turn before, and the second controller fails with `turn_binding_mismatch` or with a replay conflict. - **A slow `appendEvent`, with events that arrive behind the stored event** (3 cases, each run 3 times with the fix and 3 times without it, with the same result in each run). R7: the append of the `turn.completed` that waits is slow, and the provider-started turn starts behind it. With the snapshot at the event, the checkpoint is the waiting state at the sequence that Core has, and Core gets the `turn.started` of the turn one time. Without it, the checkpoint names the provider-started turn as active, the transport builds no start, and Core never gets that `turn.started`. R8: the first controller answers the `paperclip_finish` call while the append of the provider `turn.started` is slow. With the snapshot at the event, the checkpoint has no result and the run ends with `native_finalization_missing` (see Risks). Without it, the checkpoint has a result that Core does not have, and recovery stops with "session cannot start another turn". R6: the append of the provider `turn.started` is slow and a delta is queued behind it. With the snapshot at the event, the new controller continues the sequence that Core has. Without it, the only difference is one sequence number that Core never gets. - **The start was acknowledged before the restart** (3 cases: one turn open across the restart; two turns; a turn that waits for the answer to its `paperclip_finish` call). The new driver gets the start of each turn one time before its other events, the finish call is answered exactly one time, and the session closes with no error. - **The turn ran with no controller attached** (2 cases, with and with no result) and **two turns in one outage** (2 cases): the turns arrive in order after the recovery. - **A provider-started turn that is active in the checkpoint** gets no second start, and the next provider-started turn arrives after it. - **The turn already has its result when recovery completes** (3 cases: the turn made its `paperclip_finish` call with no controller attached; the turn waited for the answer to that call at the restart; two turns across the restart). The test asserts the exact list of events that Core gets: each provider-started turn has one start, its usage item, its message item and its terminal, and then `run.result.accepted` and `run.terminal`. `completeRun` is called one time with the turn that has the result. Before the change to the session terminal, the second controller appended only `run.result.accepted` and `run.terminal` in each of these cases. - **The wait checkpoint is not saved, then a restart** (R10). The last checkpoint is the checkpoint of the controller turn, Core has the terminal of that turn, and the start of the provider-started turn is queued. The run settles with the result of the provider-started turn, Core has one `turn.started` for it, and the finish call is answered one time. Without the wait in the driver, recovery stops with `turn_binding_mismatch`. - **An ordinary turn**, with no option. One that waits for the answer to its finish call at a restart: Core gets its result, its terminal and the run terminal one time. One whose checkpoint after `startTurn` is ahead of Core at a restart (R9): the run settles, and Core does not have only the one event that the first controller did not store. These cases pass with and without this pull request. - **No checkpoint with a terminal and no result** is saved by the two controllers in the cases above. - `src/live/provider-turn-tool-order.test.ts` and `src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests cover a stored start with removed events after it, a dropped delta, the terminal of the turn before and after the event, a terminal of a different turn after a removed terminal, and a controller start. With no option, with a different agent or provider, and for a turn that the controller started, the notifications of the transport are the same as before this pull request. - `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed. Three new tests use the real commit path: a full window of 4,096 events removes the events after a provider start and the start is found; the window removes the `turn.failed` of a provider turn after a later `turn.failed` with no start, and no start is found; a restored checkpoint that dropped the deltas of the turn, and the start is found. - `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`: 491 passed. The driver tests cover a provider start that is queued before `recoverSession()` is called, the wait state with one turn in progress at `reconcile()` (and the two cases that it must still refuse), a restored wait with a consumed disposition marker (the provider turn is admitted and `startTurn` is refused), the snapshot at the two events (also after the driver moved on, and for a restart continuation that ends waiting), one later turn in progress, one completed and a second in progress, two completed with a result, and the cases that `reconcile()` must still refuse. The backend tests cover the session terminal and the snapshot as of a sequence. - `src/native-session-runtime.provider-turn.test.ts` and `src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a restored wait with a consumed disposition marker: the provider-started turn arrives and settles the run, or the wait ends at its limit and the run takes the path of today with no new turn. - `server`: `native-session-executor.test.ts`, `native-provider-turn-wait.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21 skipped. - Mutation checks, each made named tests fail: no checkpoint at the start of a wait; no checkpoint at the start of a provider-started turn; the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in the sequence, no `turn.started` in Core, a different failure); the backend ignores the sequence; the driver has no snapshot at the event; no start-first rule; no exclusion of the active turn of the recovered driver; the turn binding only at the first delivery; the contiguous sequence required again; no check of the first terminal after the start; the adoption of later turns put back; no exception in `reconcile()`; no check after the settlement in `reconcile()`; the terminal of the session set again by a turn that ends waiting; no wait of the provider start for `reconcile()`; no exception in `reconcile()` for a session that waits; the pump deferred for a restored wait; the consumed-marker shortcut taken for a snapshot that waits. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** Each change is behind the waiting state, a provider `turn.started`, or the option of the transport. No caller sets the option before part 3. - **An event that the first controller received but Core did not store is not sent again after a restart.** This is the same on `master` and is not changed here. Only the terminal of a turn is built again. If the lost event is the result of a provider-started turn (the first controller answered the `paperclip_finish` call and stopped before Core stored the proposal), the run ends on the path of today for a turn with no result. R8 shows this. R9 shows the same loss for one event of an ordinary turn. - **A second `turn.started` for one provider-started turn can reach Core.** This occurs if the restart comes after Core stored the start and before the checkpoint of that start is saved (the time of one checkpoint write). The last checkpoint then waits, and the transport builds the start again for the new driver. Seen with the real daemon when the checkpoint write is held; not in a committed test. The usage accounting of the server uses the turn id as its key, so there is no second count, but the second start resets the "restored" mark of that turn. The transcript does not read `turn.started`. - **The older checkpoints can also save a cursor that is ahead of Core** (measured for the checkpoint after `startTurn` of an ordinary turn). Production accepts the gap in the sequence that follows, so this is not a failure there. It is on `master` and is not changed here. - **When the terminal of a provider-started turn was removed from the stored events,** the transport builds no start for an event of that turn, and the session stops on that event, as before. This needs a later terminal of the same type with no start of its own (a turn that was lost while it started) and a full window. - **The last checkpoint of a wait that ends with no provider-started turn has no terminal,** as the section above says. The change is only in a run that waits, and no run waits before part 3. - **A wait with a consumed disposition marker has no test with the real daemon.** The state needs two restarts in one run, and the test helper does one. Unit tests of the driver and of the consumer cover it. - **A provider start that waits for `reconcile()`** waits as long as the reads of `reconcile()` take. Each read ends at the command deadline of the Runner (30 seconds by default) or when the transport closes. - **`item.delta` events across a restart are not asserted.** A streamed delta that the first controller received is not sent again. The completed items are asserted. - **The driver keeps a maximum of 8 snapshots at events** and drops each one when the consumer asks for it or for a later one. If the consumer is more than 8 such events behind, the checkpoint uses the current snapshot, as before this change. - **A second provider-started turn that also ends during the same outage** is not tested with the real daemon: the fake gives each terminal the same goal revision, and the Runner then holds the second terminal until a controller is attached. The end of the reconnect time of the Runner is not tested here. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, decided the designs, reviewed the diffs, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with tool use (shell, file edits, local test runs), wrote the change and the tests, reproduced the review findings with the real daemon, and verified this slice alone. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
68d5c9c120 |
feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can already keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started (#21423, #21316). Core can already wait for that turn (#21453, #21454, #21455). No run does this yet, because no controller asks the Runner for it. > - The request goes through the transport of the Runner package. This pull request gives the transport an option for it and the behavior that the option needs. > - No caller sets the option in this pull request. It changes no production run. > - The benefit is a small base for the two pull requests after it: the recovery after a controller restart, and the switch that turns the behavior on. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. **This pull request:** the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It adds the support that parts 2 and 3 use. **Subsystem affected** Paperclip Runner package, the controller side: `src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a caller that sets the new option. **Current behavior** - The transport has no way to ask the Runner for provider-started turns. **Proposed behavior** - The transport options have a new field, `providerStartedTurns`. The transport uses it only for ACPX with the agent `claude`. In each other case the field has no effect. - With the option, the transport sends `providerStartedTurns: true` in `turn.start`. It does not send it for a compaction turn. - With the option, a tool call in a provider-started turn waits until the transport gave each earlier event to the driver. Without this, a `paperclip_finish` of that turn could reach the driver before the `turn.started` of that turn, and the driver would refuse it. - With the option, the transport gives a `run.terminal` of the Runner to the driver as a marked internal notification (`paperclip/runTerminal`). The driver uses it only during a wait, so a wait ends at once when the Runner says that the run is over. - With no option, the transport does the same as before this pull request. Live sessions, evals, the CLI runner and the probes do not set the option. **Reason and benefit** - The opt-in is a choice of the caller, not a property of the agent. Only a consumer that can wait for a provider-started turn must ask for one. `CapabilityLiveSession` completes a turn at the first `turn/completed`, so it must not get the behavior because its agent is Claude. ## What Changed - `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the option, the field in `turn.start`, the order of the tool call, and the `paperclip/runTerminal` notification. - `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the source sequence of a tool call that belongs to a provider-started turn. - `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7 cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the real consumer of Core. - `src/live/live-session.test.ts` (12 lines added): a Claude live session gives the transport no such option. - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only): the mode `provider-result-resolved`. In this mode the provider-started turn calls `paperclip_finish` and waits for the answer of Core through the Runner (`tool.resolve`), as the real sidecar does. The mode `provider-result` is not changed. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it. - **Rollback of the Runner package:** the previous transport does not send the field, so the Runner does not wait. ## Verification At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and 3: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node --test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed in each. The cases: - background work, then the turn of Claude calls `paperclip_finish`; Core answers; the Runner gives the answer to the sidecar with exactly one `tool.resolve`; the turn completes; the run finalizes with one `run.result.accepted`; - a Claude run with no option from the caller takes the path of today: no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`, no wait, and no `session.provider_turn` to the sidecar; - the limit ends the wait, the run takes the path of today, and the provider processes had exited before the cleanup of the test; - the provider process is lost during the wait, and the run settles in seconds; - the sidecar exits during the wait, and the run settles in seconds; - a turn with no background work does not wait; - an ACPX agent that is not Claude is never opted in, also when the caller sets the option. - Each case asserts the result of the close of the session: clean in 6 cases, and a named failure in the case where the sidecar is killed. Each clean case also asserts that there is no `native_session_settlement_incomplete` diagnostic and no "conflicting duplicate tool result" in the durable state. - `src/live/live-session.test.ts`, `src/live/runnerd-codex-transport.test.ts`, `src/live/runnerd-acpx-provider-turn-wait.test.ts`, `src/native-session-runtime.provider-turn.test.ts`: 434 passed. - Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102 passed). - Mutation checks, run on the full change before the split: with the option ignored, 6 of the 7 assertions of the case with no option fail; with the old mode of the fake, the case with the result fails at the close of the session. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No caller sets the option. Core sets it in part 3. - **The transport is shared by each provider.** The three new behaviors are behind one condition: the option, ACPX, and the agent `claude`. With the condition false, the code path is the path of today. - **The tool-call ordering uses a 10 ms poll** while earlier events are not delivered, only in a provider-started turn. - **A controller restart during a wait or during a provider-started turn is not handled in this part.** Part 2 handles it. Part 3 must not merge before part 2. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b1f608296e |
feat(runner): keep a run open for a limited time while the Runner waits for the provider's own turn (#21454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - `consumeTurn` reads the events of one turn and returns at the terminal. It already continues past a terminal for a Codex goal. > - This pull request makes it continue past a terminal that waits for a turn of the provider, and it limits that wait. > - The benefit is that a run continues in the turn that Claude starts, and a wait always ends. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 2. Its base branch is `feat/core-provider-turn-driver`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the consumer of a native session does after a turn that ended with no result while the Runner waits. **Subsystem affected** Paperclip Runner package, the controller side: `src/native-session-runtime.ts` and a new module `src/native-provider-turn-wait.ts`. **Current behavior** - `consumeTurn` returns at the first terminal. With no result, the run is finalized as `native_finalization_missing`. **Proposed behavior** - At a terminal with `awaitingProviderTurn` and no result, `consumeTurn` continues to read. It consumes the provider-started turn as the next turn of the same run. A turn that reports a result finalizes as a normal run. - The wait has a limit of 10 minutes from the first terminal that waits. A later terminal that waits does not start the clock again. The limit is never later than 5 seconds before the timeout of the run. - The limit and an early end stop the wait only. They never interrupt a provider-started turn that is running; that turn stays under the timeout of the run. If that turn ends and the Runner waits again, the run takes the path of today at once. - When the wait ends with no turn active, the consumer returns the last terminal, and the existing code for a run with no result decides (`resolveMissingResult`, then `native_finalization_missing`). No new failure code. - The wait ends at once when the driver gives the acknowledgement of part 1. This covers Stop, the limit, and a Runner that says the wait is over. - `executeNativeSession` gets one new option, `providerTurnWait`: the start time of a wait that an earlier controller process saw, and a hook that ends the wait early. - When the wait ends a turn, the consumer first waits for the receipts of the text deltas that are committed in batches (#21460), as each other return path of `consumeTurn` does. Without this, a delta that arrived during the wait was not in the event count of the turn, and a delta that failed to commit was not reported. - When the interrupt that ends the wait is not available or fails, the limit forces the end of the wait. If that forced end starts a new turn (the continuation that a restart owes), the consumer continues to read that turn. The forced-end signal is used one time only, and the read that is open at that moment is kept until it gets its event. Before this fix, the consumer returned the terminal of the old turn and lost the first event of the new turn, and the run could end as `native_finalization_missing`. - If the controller restarts during the wait, the run continues to wait. When the saved checkpoint still shows the first turn as active, the recovered session gives the waiting terminal again, and the wait starts again with the start time that was recorded. When the checkpoint was saved after the waiting terminal, the consumer finds that terminal in the journal, starts the wait again, and does not send the disposition-only recovery prompt. The consumer does this only if two conditions are true: the recovered session has the waiting state (`awaitingProviderTurn`, saved by the driver since #21453), and the terminal in the journal has the marker. If one condition is false, the run uses the recovery path of today. When a wait that started again ends with no turn, the existing code for a run with no result decides, as for each other wait. **Reason and benefit** - The 10 minutes are the maximum time of a foreground command in the pinned Claude Code. - A limit on the wait, not on the work, means that a long test run in the turn of Claude is not cut. ## What Changed - `src/native-provider-turn-wait.ts` (new, 129 lines) and `src/native-session-runtime.ts` (58 lines added, 4 removed), `src/index.ts` (1 export). - `src/native-session-runtime.provider-turn.test.ts`: written again for the new behavior (557 lines added, 61 removed). ## State saved by the previous release - This part adds no persisted state. The wait is in memory. Part 3 reads its start time from an event row that is already saved. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `src/native-session-runtime.provider-turn.test.ts`: 30 tests. They cover: a controller restart with the first turn still active in the checkpoint (the wait continues; a provider-started turn settles its result; with no turn the limit ends the wait on the recorded clock); a controller restart with the checkpoint saved after the waiting terminal (the same two results, no recovery prompt, no event of the journal appended again); a limit that passed before the restart ends the wait at once, with and without an interrupt; the recovery prompt of today when the terminal in the journal has no marker, when the journal has no terminal, and when the recovered session has no waiting state; a second restart during the wait; a forced end that starts a new turn settles the result of the new turn, with each event of that turn appended one time and in order; an event that reaches the open read after the forced end is appended exactly one time; a delta that arrives during the wait is counted when the limit ends the turn (the test fails when the new wait for receipts is removed); continue and settle the result of the provider-started turn; a second wait does not start the clock again; the limit with no turn active; the limit and a comment during a running turn (not interrupted); a running turn that ends and waits again after the limit; Stop; a `run.terminal` that the Runner held; the Runner stream ends during the wait; the limit under the timeout of the run; a start time from an earlier controller; a plan that a provider-started turn proposes settles as the plan review wait. - `src/native-session-runtime.test.ts`, not changed by this PR: 297 passed. - Mutation checks, each made tests fail: no continue at a waiting terminal; no limit; no early end; no clock from an earlier controller; the clock starts again at each wait; a running turn is interrupted at the limit. - Mutation checks for the forced end, each made the two new tests fail: the forced-end signal used on each loop (the run ends as `native_finalization_missing`); a new read started after the forced end (the first event of the new turn is lost). - Mutation checks for the restart, each made named tests fail: the recovery prompt not held back; the wait not started again in `consumeTurn`; the wait started again on the session state alone (the test with no marker in the journal fails). Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No terminal has the marker before part 4. - **`consumeTurn` runs for each native turn of each agent.** It makes one small object for each turn and reads one more payload field at a terminal. - **If the interrupt that ends the wait is not available or fails,** the limit forces the end: the consumer returns the last terminal, or continues to read when the forced end starts a new turn. This is a fallback path. On a forced return, one read of the event stream stays open until the session closes, as before this PR. - **A wait that starts again after a restart holds back the recovery prompt.** This is the one change to the recovery flow. It needs the two conditions above, and both exist only for a run that asked for provider-started turns. - **Known limit.** The driver does not clear its waiting state when the consumer ends a wait with an interrupt. A restart in the short time after that can start the wait again; the recorded clock and the comment check then end it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
82ce8e64ba |
feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Core reads Runner events through the Codex app-server driver, also for ACPX agents. The driver refuses a turn that the controller did not start, except a goal turn of Codex. > - This pull request lets the driver accept a turn that the provider started, only directly after a turn that the Runner marked as waiting. > - The benefit is that the next parts can use that turn. Nothing reaches this code until the last part sends the opt-in. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 1. Its base branch is `feat/adopt-claude-provider-started-turn`. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. What the driver does with a turn that the provider started. **Subsystem affected** Paperclip Runner package, the controller side: `src/drivers/codex` (the driver that Core uses to read a native session). **Current behavior** - A `turn/started` that the controller did not ask for fails the session with `turn_binding_mismatch`, except for an active Codex goal. - An interrupt with no active turn fails with "already terminal". **Proposed behavior** - A turn terminal that the Runner marked with `awaitingProviderTurn` and that has no result puts the driver in a waiting state. The driver event `turn.completed` carries the marker. - In that state, a `turn/started` with `origin: "provider"` is admitted the same way as a goal turn, and the driver event carries `origin: "provider"`. In each other state it still fails closed. - In that state with no active turn, an interrupt gives an acknowledgement item with the status `provider_turn_wait_ended`. It sends nothing to the Runner. The driver also gives that acknowledgement when it gets the internal `paperclip/runTerminal` notification with a private marker; a notification without the marker does nothing. - An interrupt for a provider-started turn that is running is sent to the Runner only when it names that turn. - The driver saves the waiting state in the persisted session (`awaitingProviderTurn`, written only during a wait). After a controller restart, the restored session still admits the provider-started turn and still acknowledges an interrupt during the wait. The driver restores the state only when the session has no active turn, no stored result and at least one finished turn, so a new run cannot take the wait of the run before it. - When a restart finds that the saved active turn is finished, the turn read of the transport and `reconcile()` of the driver carry the same marker as the live `turn/completed` notification. `runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from the saved `turn.completed` event to the turn object of a read. - The driver ends the waiting state when the Runner reports the run terminal. **Reason and benefit** The consumer in the next part needs these three signals: a marked terminal, an admitted turn, and a clear end of the wait. ## What Changed - `codex-session-terminal.ts`, `codex-session-notifications.ts`, `codex-session-state.ts`, `codex-harness-session.ts`, `codex-driver-values.ts`: 37 lines added, 4 removed. - `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines). - `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`, `backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`: the optional field in the two persisted shapes and its copy in each direction. - `live/runnerd-codex-transport.ts`: the marker on a recovered turn terminal. The loop that builds the earlier terminals of a `thread/read` is now the function `runnerdPriorTurnTerminals`, with the same logic, so that a test can call it. ## State saved by the previous release - One new optional field in the persisted driver session: `awaitingProviderTurn`. A snapshot outside a wait has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so the driver does not wait. This is the behavior of today. - **State from this release, read by the previous release:** the previous reader copies the fields that it knows and ignores this one. No reader rejects it, and the integrity digest covers the stored object as written. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - 6 driver tests: a provider-started turn after a marked terminal is admitted and can report a result; a provider-started turn with no marked terminal fails closed; an interrupt during the wait is acknowledged; a running provider-started turn is not interrupted by the end of the wait; the marked notification ends the wait and a notification with no marker does not; a normal terminal is not changed. - Restart tests: a session restored from a snapshot with the waiting state admits the provider-started turn and settles its result; a snapshot outside a wait has no new field; a restored session with no waiting state refuses a provider-started turn as today; an interrupt during a restored wait is acknowledged. A saved active turn that the read reports finished with the marker gives a `turn.completed` with `awaitingProviderTurn: true`; with no marker it is an ordinary terminal. - `harness-driver-backend.test.ts`: the field goes through a serialized checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324 passed): the marker is on a recovered terminal only for a `turn.completed` with a boolean `true`. - `src/native-session-runtime.test.ts`: 297 passed, with the real `paperclip-runnerd`. - Mutation checks for the restart work, each made a named test fail: no restore of the state; no copy in the backend; no copy in the transport; no marker argument in `reconcile()`; no end of the wait at the run terminal. - Mutation checks: the acknowledgement branch removed, the admission removed, and the marker check removed each made a test fail. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** The Runner sends the marker only after the opt-in of part 4. - **The admission rule is shared with Codex goal turns.** The existing goal tests pass with no change. - **Known limit after a controller restart.** If the Runner already runs a provider-started turn and the saved active turn is the turn before it, recovery fails closed, as today for each unexpected active turn. - **A controller turn that starts during a wait** clears the waiting state when its `turn/started` arrives, not when it is sent. A snapshot in that short time can hold an active turn and the field together; the restore rule drops the field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f7b1651799 |
feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The parts before this one give the adapter an ordered stream of those cycles (#21421), declare the contract (#21422), and make `paperclip-runnerd` wait for such a turn (#21423). > - This pull request is the last part in the Runner. The sidecar reads the stream, and gives a cycle to `paperclip-runnerd` as a provider-started turn only when `paperclip-runnerd` armed it. > - The benefit is that a later change in Core can let a run continue when background work finishes. Until Core sends the opt-in, a production run does the same as today. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request was first opened with the full change (49 files). It is now split. The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. This pull request: the sidecar adopts a cycle. Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code at the head of this branch is the same as before the split; only the base changed. Review comments made before the split are about code that is now in the parts above. Refs #19912 (design). **What existing behavior does this improve?** What the ACPX sidecar does with a cycle that Claude starts by itself. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only. **Current behavior** - The sidecar does not read the cycle stream of the adapter (#21421), so each cycle is declined and the background state becomes unknown. - The sidecar does not know the command `session.provider_turn`, so `paperclip-runnerd` (#21423) never waits. **Proposed behavior** - The command `session.provider_turn` with `armed: true` arms the sidecar for one turn. The sidecar accepts it only for the terminal that reported a running background shell, with no active turn and a provider that is alive. One flag holds the arming. It is cleared at each turn start and turn end, on suspend and close, and when the provider is lost. - One loop, `consumeProviderCycles`, reads the stream. For each cycle: - if the sidecar is armed and no turn is active, the sidecar adopts the cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`, then the events, the usage and the terminal of the cycle, as for a normal turn; - if the cycle cannot be decided yet (the prompt turn is still settling, or `paperclip-runnerd` did not arm yet), the sidecar holds it for a maximum of 5 seconds, and then adopts or drops it; - in each other case the sidecar drops the cycle. The stream ends, and the background state becomes unknown, so `paperclip-runnerd` restarts the provider at the end of the turn as today. - A Paperclip tool call or a permission request of a held cycle waits for the decision of its own cycle. It is never bound to the prompt turn or to a different cycle. - If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar publishes the same `plan.proposed` event as for a controller turn, and the turn ends with the same terminal: completed, with no `backgroundTasks`, so `paperclip-runnerd` does not wait again. A controller turn is not changed. - If the provider exits while the sidecar is armed with no turn active, the sidecar sends a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait. - `turn.cancel` for a provider-started turn cancels the cycle and closes the provider. - With no arming, nothing is shown. No controller sends the opt-in yet, so `paperclip-runnerd` never arms. **Reason and benefit** - Only `paperclip-runnerd` decides that a cycle becomes a turn. The sidecar cannot start a turn by itself. - One reader of one ordered stream means that two cycles cannot overlap. ## What Changed - `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the arming flag, `consumeProviderCycles`, `adoptProviderCycle`, `pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is not decided. - `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines) and two fake hosts in `src/test-support` (227 lines). One host follows a script. The second runs the real cycle stream and the real background task tracker. - `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter for a test harness. No schema, no Rust and no persisted state changes in this part. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `pnpm run typecheck:typescript`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript`: no errors. The server check `node scripts/verify-runner-vendor-dependencies.mjs`, the cache guard and the contract tests: passed. - `npx vitest run src/cli src/drivers/acpx src/live/runnerd-acpx-provider-turn-wait.test.ts src/drivers/runner-tool-bridge.test.ts src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)` files, `2147 passed`, 9 skipped. - `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites: all passed. No Rust changed in this part. - The sidecar tests of this part: a cycle that starts before the arming command, or while the prompt turn settles, is held and then adopted with its tool call; a cycle is never shown when `paperclip-runnerd` did not arm, disarmed, armed for a different terminal, armed for a quiet terminal, or armed after the provider exited; two cycles in sequence with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a lost provider; Codex gets no change. - Mutation checks, each made the named tests fail: the sidecar arms with no matching terminal; no hold; the old hold condition that drops a cycle while the prompt turn settles; no wait of a tool call for the decision. - Live run with a real Claude account, not in CI, with the real ACP runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the adapter level: one background shell gave `{ live: 1 }`, one cycle with its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were admitted, and none was refused. Not run: a live run through the real sidecar process, `paperclip-runnerd` and Core (that path needs an API key, not a local sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in, so the sidecar is never armed. The new code that runs today is the loop that reads the stream: with no arming it drops each cycle, and the background state becomes unknown as before. - **A follow-up for a command that finishes while Claude reports on an earlier one is lost.** The Runner waits again only while a background command is still running. The run ends as it does today. The reason is in #21421. - **A cycle that starts in the last second of a prompt turn** can have its first messages in the prompt turn and the rest in the provider-started turn. Tool calls stay safe: the admission rule of #21278 refuses a call that the active turn did not announce. - **The 5 second hold** is not tested under load. A cycle that is held longer is dropped, and the run takes the path of today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
4a78124a3e |
feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn that ended with no result while a background shell is running, this stops the shell and the run fails. > - #21278 binds each tool call to its turn, and #21293 reports how many background shells are running at the end of a turn. Part 4 declared the contract. > - This pull request makes `paperclip-runnerd` wait for a turn that Claude starts by itself, when the controller asks for it, and admit that turn into the same run. > - The benefit is that the Runner has one place that decides to wait and one place that ends a wait. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 5. Its base branch is `feat/prp-provider-turn-contract`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** What `paperclip-runnerd` does after a Claude turn ends with no result while a background shell is running. Nothing changes until a controller sends the opt-in. **Subsystem affected** Paperclip Runner: the ACPX provider backend in `runner-core` (Rust). Claude only. **Current behavior** - After each completed turn the Runner emits `run.terminal`. A turn with a running background shell gets the provider restart, and the shell stops. **Proposed behavior** Nothing changes unless the controller sends `providerStartedTurns: true` in the payload of `turn.start`. No controller sends it yet. With that field: - The Runner keeps the provider and waits when all of these are true: the agent is Claude; the turn completed; the run has no accepted `paperclip_finish` or `paperclip_block` result; the terminal event reported `backgroundTasks` with `live` above zero; each condition of #21293 for a kept process is true; the sidecar accepted `session.provider_turn` with `armed: true`. - `turn.completed` then has `awaitingProviderTurn: true`, and no `run.terminal` follows. The Runner holds that `run.terminal`. - When the sidecar reports a turn that the provider started, the Runner admits it only during a wait and publishes `turn.started` with `origin: "provider"`. In each other case that report fails the session closed. - When a wait ends with no turn active, the Runner publishes the held `run.terminal`. This applies when the provider process exits, the sidecar exits, the controller cancels or stops the run, the session is closed or suspended, and after a Runner restart. - The held `run.terminal` and the removal of the wait are saved in one write, and the controller gets the event from the next poll. A crash after a cancel or a stop cannot lose it, and it is never sent two times. - A controller `turn.start` during a wait replaces the provider first, as before, and the held event is dropped. - A `run.attach` from a different run during a wait ends the wait when the attach succeeds. The Runner then discards the held `run.terminal`, because that event is of the old run and must not end the new run. This is the same rule that a Runner restart applies. The wait is removed only in the state that the attach saves for the new run. If the attach fails, for example when the old session does not stop, the old run keeps the wait and gets its held `run.terminal` one time, from the next poll or from a cancel or a stop. A `run.attach` from the same run publishes the held event to that run. - The Runner waits at most 5 seconds for the sidecar to answer a request to arm or to disarm a provider-started turn. This is the limit that turn cancellation already uses. Before, the limit was the 30 seconds of an ordinary request, and a cancel, a stop or a shutdown had to wait behind it. No answer is handled as a lost sidecar: the session fails closed, and the Runner still publishes the held `run.terminal` one time. - The sidecar of this branch does not know `session.provider_turn`, so the arming request fails and the Runner does not wait. The behavior comes with part 6. **Reason and benefit** - The wait is one entry in the saved session state. One rule ends it at each save, so each new way to leave the open session ends the wait with no new code. - A wait must not outlive the provider or the sidecar, and a cancel must stop it. ## What Changed - `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests): `provider_turn_wait { turn_id, held_run_terminal }` in the saved state; the rule that ends a wait at save; `settle_turn_events`, which decides at one place to arm, to create the wait and to put `awaitingProviderTurn` on `turn.completed`. - `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its terminal, the arming request, and `admit_provider_turn`. - `acpx_provider_session.rs`, `acpx_provider_backend.rs`, `acpx_provider_state.rs`, `provider_events.rs`, `acpx_park_preparation.rs`: the opt-in from `turn.start`; the record that the run has a result; the provider-started turn in the state and in the PRP events; cancel and stop with no turn; no turn-end restart and no park during a wait. - `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no event also checks that the sidecar process is alive. A child process can keep the output stream of a dead sidecar open. - `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and failures for tests. - Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`, `tests/support/acpx_quiet_turn_restart.rs`, `src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon), `src/native-session-runtime.provider-turn.test.ts` (the controller of today). - `runner/DURABLE_TRANSPORT.md`: the rules. ## State saved by the previous release - One new optional field in the saved ACPX session state: `providerTurnWait`. It is written only during a wait. In each other state the saved file has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so there is no wait. A unit test covers this. - **State from this release, read by the previous release (a rollback):** outside a wait, the file loads as before. During a wait, the previous Runner refuses the file with its "ACPX provider state is malformed" error, because its loader refuses each unknown field. Only a run that waits at the moment of a rollback is in this case, and no run waits before a controller sends the opt-in. - **Runner restart during a wait:** the wait is restored. The provider is a new process, so the wait is over, and the controller receives the held `run.terminal` one time. A test with the real daemon covers this. - **Controller of today or of the previous release:** `turn.start` has no opt-in. A test runs a turn with a running shell and a turn with an unknown background state, and compares the events, the commands after the turn, the restart, and the park state before and after the restart. They are the same. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored. All 17 `acpx_*` suites passed. - `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends with exactly one `run.terminal` when the sidecar exits while a child process keeps its output open, when the provider is lost, when the controller cancels, and after a Runner restart; with no opt-in, `run.terminal` follows the turn and the sidecar is never armed. - Rust tests: wait and admit; no wait with a result; no wait with no opt-in; an active goal neither marks nor arms a wait; a sidecar that refuses to arm; a provider-started turn that the Runner did not wait for fails the session closed; cancel and stop during a wait; provider and sidecar loss; restart; the saved shape in both directions. - `src/native-session-runtime.provider-turn.test.ts`: the real `consumeTurn` of today gets the event sequence with no opt-in and finalizes it as `native_finalization_missing`, as today. - Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a different run attaches after the snapshot reports `warmAttachReady: true`; the new run gets no `run.terminal`, the saved state has no wait, and the old session is closed. A same-run attach gives exactly one `run.terminal`. An attach that fails when the old sidecar is dead leaves the wait with the old run, and the old run then gets exactly one `run.terminal` from a poll, `run.cancel`, `turn.interrupt` or `turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in the pending events of the new run; with the wait ended before the attach is complete, the old run gets no `run.terminal` after a failed attach. - No answer from the sidecar, in `tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not answer the disarm request; `run.cancel` and `turn.interrupt` each return `already_settled` in less than 10 seconds, exactly one `run.terminal` reaches the run, `session.close` completes in less than 2 seconds with no second `run.terminal`, and the provider process is gone. With no answer to the arm request, the turn ends with its `run.terminal` and with no wait. With the limit removed, each test fails at 30 seconds. - Mutation checks, each made the named tests fail: the rule that ends a wait at save; no `run.terminal` after a restart; arming with no opt-in; the marker set with no wait; the sidecar process check removed. Not run: a rollback with the binary of the previous release; a run with the opt-in and a sidecar that answers the arming request with an error (the code reads each error as not armed); Linux; the full suite. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in. - **A rollback of the Runner during a wait** makes the previous Runner refuse that saved session. See "State saved by the previous release". - **The wait has no time limit in the Runner.** The controller must close, suspend or cancel. Core will own that limit. - **Park is refused during a wait.** The controller must end the wait before it parks. - **A new run can attach during a wait.** The attach is not blocked, because the Runner has no time limit on the wait and a new run can send only `run.attach`. After an attach that succeeds, the old run gets no `run.terminal` from the Runner; the controller must not depend on that event. The request that disarms a sidecar that the attach keeps has no test, because no test can keep a session across an attach during a wait. - **After a sidecar that does not answer, `runner.suspend` returns an error at once** ("provider lifetime cleanup is not yet proven"), and `session.close` completes. This is the behavior of today for each lost sidecar; this PR does not change it. - **The shared code that changed** runs for each Claude turn: the record of a run result, and the settle step at each turn end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e82c4f0a31 |
fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**
This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it
## Linked Issues or Issue Description
**What happened?**
Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.
- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.
**Expected behavior**
A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.
**Steps to reproduce**
1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.
**Paperclip version or commit**
`master` at
|
||
|
|
1332ab987f |
feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The Runner, its sidecar and the controller speak two protocols with schemas: PRP between the controller and the Runner, and the sidecar protocol between `paperclip-runnerd` and the ACPX sidecar. > - A turn that the provider starts needs new fields and one new command in those protocols. > - This pull request declares them and generates the readers again. It changes no behavior. > - The benefit is that the next two pull requests show behavior only, with the contract already in review. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 4. Its base branch is `feat/claude-provider-cycle-stream`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production. The protocol schemas. **Subsystem affected** Paperclip Runner: `protocol/` and the generated TypeScript and Rust contract code. **Current behavior** - PRP has no way to say that a turn was started by the provider, or that the Runner waits for such a turn. - The sidecar protocol has no command to arm the sidecar and no event for a turn that the provider starts. **Proposed behavior** - PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns` (boolean) in the payload of `turn.start`, `origin: "provider"` on `turn.started`, and `awaitingProviderTurn: true` on `turn.completed`. `command-v3` and `event-v3` refer to it. - Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the command `session.provider_turn` with `{ turnId, armed }`, the event `runtime.turn_started` with `{ origin: "provider" }`, and the payload of the diagnostic for a lost provider. The schema enforces that payload for a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`. Diagnostics with other codes keep their open payload. - `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No sidecar sends it yet. The Runner of today refuses an unknown event too. - No controller, Runner or sidecar uses the new fields in this pull request. **Reason and benefit** The repository rule for Runner contracts says that new fields must be in the schema with generated readers. A separate pull request lets a reviewer read the contract alone. ## What Changed - `protocol/schemas/provider-turn.schema.json` (new), `command-v3.schema.json`, `event-v3.schema.json`, `protocol/provider-schemas/acpx-sidecar.schema.json`, `protocol/manifest.json`. - `scripts/generate-acpx-sidecar-contract.mjs` and `scripts/generate-protocol-schema-module.mjs`. - Generated: `src/drivers/acpx/generated-sidecar-contract.ts`, `src/protocol/generated/schema-bundle.ts`, `src/protocol/generated/standalone-validators.ts`, `runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`. - `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the Rust `match` on the event type must name the new event to compile. - `test/protocol-contract.test.mjs`. - The contract digests of the pack change (`prp.acpx-sidecar`, `prp.command-v3`, `prp.event-v3`, `rust`, and the new `prp.provider-turn`). No stored file refers to these digests. No provider `commandDigest` and no profile version changes. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. The contract tests: 32 passed. - The sidecar suite and the tracker tests: 141 passed. Not run: the full suite. CI supplies that coverage. ## Risks - **The pack contract digest changes.** Older image packs are not used again. - **The opt-in field is accepted by the schema but has no effect yet.** A controller that sends it before part 5 gets the behavior of today. - **State saved by the previous release:** this pull request changes no persisted shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e5ad224cc5 |
feat(acpx): give Claude's own cycles to the adapter as one ordered stream (#21421)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - Claude Code starts a cycle of its own when a background task finishes and no prompt is in flight. The Runner sees the raw messages of that cycle. > - #21293 only marks the background state as unknown when it sees such activity. > - This pull request gives each such cycle to the caller of the adapter as one ordered stream: started, its events, ended with its result. Nothing reads the stream yet. > - The benefit is one place that knows where a cycle starts and ends, with cycles that cannot overlap. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 3. Its base branch is `feat/acp-session-update-event-converter`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production yet. How the ACPX adapter sees the cycles that Claude starts by itself. **Subsystem affected** Paperclip Runner: `src/drivers/acpx` (the adapter, the runtime host, the Claude background task tracker). Claude only. **Current behavior** - The raw ACP observer gives each message to the background task tracker. - Activity of Claude with no client request in flight makes the background state unknown. The messages of that cycle are not kept. **Proposed behavior** - `ClaudeProviderCycles` puts each cycle on one ordered stream. A cycle starts at the first update of the own session with no client request and no prompt in flight. It ends at the `usage_update` that has `_meta._claude/origin`. - A caller gets the stream with `providerCycles()`, reads it with `next()`, and calls `adopt()` or `drop()` for each cycle. A permission request waits for the decision of its own cycle. The stream records the cycle of each tool call, and `decisionFor(toolCallId)` gives that decision. The binding that a caller gives to `adopt()` can also carry `onPlanProposal`, so a plan that an adopted cycle proposes goes to the caller as it does for a normal turn. - With no caller, a cycle is declined and the background state becomes unknown, as before. The sidecar does not call `providerCycles()` in this pull request, so production does the same as today. - The tracker reports `{ live: N }` only while N shells are running, `{ live: 0 }` only if the process never had a background task and Claude never acted between prompts, and unknown in each other case. A dropped, failed or cancelled cycle makes the state unknown. - An empty result that directly follows the end of a cycle (zero usage, same context use and cost) is ignored. Claude Code sometimes sends it. It must not become a cycle, and its usage must not be reported two times. **Reason and benefit** - One producer and one reader make the order of cycles a property of the structure. - The tracker does not read the order of messages to decide which finished task a cycle reported on. Captures showed that Claude Code can put two notifications in one cycle, start a second cycle, or give a notification to a cycle in progress, and the ACP stream does not say which. ## What Changed - `src/drivers/acpx/claude-provider-turns.ts` (new, 303 lines): `ClaudeProviderCycles`. - `acpx-runtime-adapter.ts` (78 added, 8 removed) and `runtime-host.ts` (24 added): `providerCycles()` and `providerTurnsAvailable()`, and the permission path that waits for the decision. - `claude-background-tasks.ts` (16 added, 7 removed): the rule above. - Tests (764 lines): `claude-provider-turns.test.ts`, `claude-provider-turn-tasks.test.ts`, `claude-background-tasks.test.ts`, a permission test in `acpx-runtime-adapter.test.ts`, two tests with the real `claude-agent-acp` (`acpx-runtime-adapter.claude-provider-turn.test.ts`, `acpx-runtime-adapter.claude-overlapping-shells.test.ts`), and a captured raw stream as a fixture. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - 10 test files, 294 passed. This includes the test of #21293 with the real `claude-agent-acp`, not changed: with no caller of the stream it still gets `{ live: 1 }` at the end of the turn and an unknown state after the cycle of Claude. - Real `claude-agent-acp` with a local fake Anthropic API: a background shell finishes, and the stream gives one cycle with its own usage. With two shells where the second finishes before the first cycle sends its first update, the first cycle ends with an unknown state. - Mutation checks, each made the named tests fail: an end item before the last message of a cycle; a drop that does not make the state unknown; a finished task that reports `{ live: 0 }`; the empty result not ignored; a permission request that does not wait. - Live run with a real Claude account, not in CI, on the full change: one background shell gave `{ live: 1 }`, one cycle, then unknown. Two shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown. Not run: the full suite. CI supplies that coverage. ## Risks - **The time at which the state becomes unknown changes a little.** With no caller, a cycle makes the state unknown at its first message. Before, it was at the first message of an activity kind. The result for the Runner is the same: a restart at the end of the turn. - **Limit, by design.** After a cycle, `{ live: N }` counts only running shells. A command that finishes while Claude already reports on an earlier one gets no cycle that the Runner waits for. - **The rule for the empty result is narrow** and comes from captures of one Claude Code version. The reason that Claude Code sends it is not known. - **A queue of 4,096 messages for one cycle.** A cycle that is larger fails, and the state becomes unknown. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1d112a504f |
feat(acpx): convert raw Claude session updates the way ACPX does (#21420)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The Runner sees each raw ACP message before ACPX does. A cycle that Claude starts by itself is not a part of an ACPX prompt, so ACPX gives no events for it. > - To show that cycle in the same form as a normal turn, the Runner must convert the raw messages itself. ACPX 0.22 does not export its converter. > - This pull request adds a copy of that converter with a test that compares it with ACPX. Nothing uses it yet. > - The benefit is that a later pull request can show the cycle of Claude with the same events as a normal turn. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 2. Its base branch is `refactor/acpx-sidecar-runtime-event-step`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing yet. It adds a module that a later part uses. **Subsystem affected** Paperclip Runner: `src/drivers/acpx`. One new direct dependency in the Runner and the server. **Current behavior** - ACPX converts `session/update` messages to runtime events inside a prompt. The function is private. - The Runner has no way to make the same events from a raw message. **Proposed behavior** - `acp-session-update-event.ts` converts a raw `session/update` to the event that ACPX 0.22 makes for the same update. - It validates the message with `@agentclientprotocol/sdk` 1.9.0. This is the version that ACPX 0.22 already installs. - No production code imports the module in this pull request. **Reason and benefit** A copy with a comparison test is a smaller risk than a patch of ACPX. The test fails when the two differ. ## What Changed - `src/drivers/acpx/acp-session-update-event.ts` (new, 367 lines). - `acp-session-update-event.test.ts` and `acpx-runtime-adapter.claude-session-update-events.test.ts` (new, 363 lines). - `packages/paperclip-runner/package.json` and `server/package.json`: `@agentclientprotocol/sdk` 1.9.0 as a direct dependency. The server needs the line because it copies the Runner build with no `node_modules`. - `pnpm-lock.yaml` is not in this pull request. The lockfile workflow owns it. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - The two new test files and the sidecar suite: 130 passed. - The comparison test runs the real ACPX runtime with the pinned `claude-agent-acp` and a local fake Anthropic API, and gives the same raw updates to the copy. The events are the same. - A second test fails if the SDK version of the Runner is not the version that ACPX resolves. With 1.4.0 in `package.json` it fails. Not run: the full suite. CI supplies that coverage. ## Risks - **A new direct dependency.** It is already installed through ACPX, so no new package is downloaded. The version test keeps the two the same. - **The converter is a copy of ACPX code.** An ACPX upgrade must run the comparison test. - **The module is not used yet.** It is used in part 3. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
055a0a3c54 |
refactor(acpx-sidecar): extract the runtime event step and the test process helper (#21419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The last part of that stack adds a second kind of turn to the sidecar. It needs two pieces of the code of a normal turn. > - This pull request moves those two pieces into functions, and moves a test helper to a file of its own. It changes no behavior. > - The benefit is that the later pull request shows only new behavior, not moved code. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 1. Its base branch is `refactor/remove-direct-gateway-path`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing for a user. The structure of the ACPX sidecar code. **Subsystem affected** Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts` and its test. **Current behavior** - `pumpTurn` has the code that prepares each runtime event for the Runner, and the code that sends the terminal frame of a turn, inline. - The sidecar test file has the `SidecarProcess` helper that starts a sidecar process. **Proposed behavior** - `createRuntimeEventStep` has the event code. `emitTurnTerminal` has the terminal code. `pumpTurn` calls both. - `src/test-support/acpx-sidecar-process.ts` has the helper, so a second test file can use it. - No protocol, event or timing change. **Reason and benefit** A later pull request adds a turn that Claude starts by itself. It uses the same two functions, and its tests use the same helper. ## What Changed - `src/cli/acpx-runtime-sidecar.ts`: 74 lines added and 56 removed. The code moved with its comments, with no change. - `src/cli/acpx-runtime-sidecar.test.ts` and `src/test-support/acpx-sidecar-process.ts`: the helper moved, 138 lines added and 132 removed. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `npx vitest run src/cli/acpx-runtime-sidecar.test.ts`: 125 passed. The tests are the same as before, apart from the import of the helper. Not run: the full suite. CI supplies that coverage. ## Risks - **A refactor of the turn loop of each ACPX agent.** The moved code runs for Claude, Codex, Pi, Kimi, Cursor, Copilot and Grok. The existing sidecar suite covers it. No new test is necessary, because no behavior changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
342fb308bd |
perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - Text deltas (`item.delta`) are about 85% of that process's SQL statements: each delta is one transaction of four statements on the shared 10-connection pool > - At 50 agents and 20 deltas per second, the process falls behind and unrelated agent requests wait for the pool > - This pull request commits text deltas in batches: one transaction for all deltas that arrive within 250 ms, with one multi-row insert > - The benefit is about 7 times fewer transactions and 5–6 times fewer statements per delta, with no delay for the first text of an item ## Linked Issues or Issue Description Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a of the agent scaling plan in #21418. **What existing behavior does this improve?** The persistence of native text deltas. `native-session-runtime` awaited `PaperclipControlPlanePort.appendEvent` for each event, and the port ran one transaction per event. **Subsystem affected** Server run event ingest (`paperclip-control-plane-port.ts`, `heartbeat-run-events.ts`) and the runner package's turn consumer (`native-session-runtime.ts`). **Current behavior** Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In the load model with 50 agents (deltas only), that is about 4.0 statements and 1.0 transaction per delta. **Proposed behavior** The turn consumer starts the next append without waiting for a delta's receipt. The port collects deltas and commits them in one transaction with one insert. In the same load model, that is about 0.7 statements and 0.14 transactions per delta. **Reason and benefit** Fewer transactions free the shared pool and the main thread for other agents' requests. This is the largest per-agent cost in the scaling plan. **Breaking changes** None. The row shape does not change. A port without the new capability keeps the earlier serial behavior. ## What Changed - `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends native source events of one run and one source in one transaction. When the events continue the stored source cursor (from #21435), it sends four statements for the whole batch: the run lock, one replay probe, one `insert ... select from jsonb_array_elements(...)` of every row, and the commit. Any other batch (a replay, a gap, an event with a wake, or a run without a cursor) appends its events one by one in the same transaction, with the single-append checks. - **Storage choice: one row per delta, written by one statement per batch.** I did not store one row per batch. The transcript, live events, run log export, chat progress, replay (`replayEvents`), recovery cursors and the UI all read one row per delta. One statement per batch keeps every reader unchanged, and rows written before this change read the same. - `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An `item.delta` appended with the new `streaming` option goes into a queue: - It commits at once until a delta with text commits after the latest other event. So the first text of each item, and the time to first token, never wait for the timer. - Later deltas commit after at most 250 ms, or at once when the batch has 100 deltas. - Every other event, `checkpointSession`, `completeRun`, `replayEvents` and `provesNoRunProviderWork` first commit the queue. A non-streaming event is never committed before the deltas that preceded it. - A delta appended without `streaming` commits at once, so callers that wait for each receipt are not slowed. - The receipt of a delta settles only after its batch commits and its observers (live event, executor callback) run, in order. - If a batch fails, every delta in it rejects and nothing from it is stored. - `native-session-runtime` `consumeTurn` appends deltas with `streaming` and does not wait for their receipts (at most 256 pending). Before any other event, any result, and the governed-wait settlement, it calls `flushStreamingEvents()` and then waits for all pending receipts, so a boundary never waits for the timer. A failed delta stops the turn before a later event is appended. The consumer joins every pending delta append before it settles, also when the provider stream fails. - If one observer of a committed batch fails, only that delta's receipt rejects. The other rows are still published and observed. - `ControlPlanePort` contract: optional `pipelinesStreamingEvents`, `flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`. ## Verification - Red/green. The new tests failed before the change and pass after it: - Runner: "starts text delta appends without waiting and flushes them before any other event" (`native-session-runtime.test.ts`). The port in the test has no timer, so the turn finishes only if the boundary flushes. Before: the appends were serial, and later the boundary waited for the timer. - Runner: "joins pending delta appends before a failed turn settles". Before: the turn settled while a delta commit still ran. - Port: "observes every committed delta of a batch when one observer fails". - Port: "commits the first text delta at once and later deltas in one batch before the next other event". It checks one transaction and one insert for five deltas, flush before a non-streaming event, receipts only after commit, and immediate commit of the first delta with text. Before: each delta committed alone. - Port: "flushes queued deltas on the timer and before a replay read, and rejects every delta of a failed batch". It checks the crash-before-commit case: a revoked writer fails the batch, nothing is stored, and the resent deltas commit idempotently (the stored one is a duplicate). - Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`: one lock, one probe and one insert per batch; replay idempotency and payload conflict; a batch that fails before commit stores nothing and its resend commits; the one-by-one path for no cursor and for gaps. - Focused suites pass: `heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`, `paperclip-control-plane-port` (including the package conformance suite), `paperclip-semantic-conformance`, `native-observer-replay`, `native-event-boundary-golden`, `live-recovery-work`, `native-interaction-bridge`, `chat-native-progress-work`, and `native-session-runtime.test.ts` in the runner package. - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript` and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418, `AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls the store directly, so for this run I added a local `BATCH_MS` mode that applies the port's batching rules (first delta at once, 250 ms timer, 100 cap, flush before other events) and does not wait for delta receipts. That mode is not in this pull request. The machine was very busy (load average 40–60), so absolute figures are noisy; per-delta ratios are the stable result: | Deltas only | Statements/s | Transactions/s | Deltas done of offered | Statements per delta | Transactions per delta | Lateness p50 / p99 | | --- | --- | --- | --- | --- | --- | --- | | One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 | 1.00 | 3,074 / 5,384 ms | | Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 / 2,858 ms | | One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 | 3.99 | 1.00 | 6,558 / 11,464 ms | | Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17 | 187 / 556 ms | With all sources on, the machine was saturated (event-loop utilization 0.96–0.98) in both modes and the results were not usable (764 and 12,800 of 20,000 deltas done). ## Risks - **Ack and crash (read first).** In the live runnerd path the runner's ack does not wait for the `heartbeat_run_events` row today. `DurablePrpControlPlane.#event` acks after the in-memory journal commit. Streaming events are not in the controller checkpoint. So a delta that is acked but not yet appended is lost if the server crashes, and the runner does not resend it. This is the earlier behavior. This pull request makes that window up to 250 ms longer. The final text still survives through `item.completed`, which commits immediately and in order. This pull request does not change `durable-prp-control-plane.ts`. - Live text in the UI arrives in steps of up to 250 ms after the first text of an item. - A delta failure now surfaces at the next non-delta event or at the end of the turn, not at the delta. The turn stops before it appends a later event. - The governed-wait drain and the control-event replay still append one event at a time. They do not carry deltas. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0aaab644fe |
refactor(runner): remove the direct assigned-gateway path (#21311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent reaches the tools of its app connections through an MCP gateway. > - Before #21274, some native harnesses got the gateway URL and a bearer token and connected to the gateway directly. The Runner has code for that: an HTTP MCP server for ACPX providers, a server entry in the Codex `config.toml`, a server entry in the OpenCode config, and the lists that pass the three `PAPERCLIP_NATIVE_MCP_*` variables to a sidecar. > - #21274 gives each native harness its gateway tools through the PRP relay. Core removes the three variables from each Runner environment. The direct code path gets no binding. > - Code that cannot run still needs maintenance, and it can still put a token in a provider process if a variable arrives by another way. > - This pull request deletes the direct path from the Runner. > - The benefit is one path for assigned gateway tools, and no Runner code that can give a gateway token to a provider. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is in a stack. Its base branch is `fix/keep-quiet-claude-provider-between-turns` (#21293). It depends on #21274, which is the bottom of the stack. On `master` without #21274, the direct path is still in use. **What existing behavior does this improve?** The Runner code that connects a provider directly to the assigned MCP gateway. **Subsystem affected** Paperclip Runner: the TypeScript drivers and the Rust `runner-core` environment lists. **Current behavior** After #21274, Core never gives `PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` or `PAPERCLIP_NATIVE_MCP_TOKEN` to a Runner. The Runner still has the code that reads them and configures a direct gateway server for ACPX providers, Codex and OpenCode. **Proposed behavior** The Runner has no code that reads the three variables to configure a provider. If the variables are in the environment of a Runner, no provider environment, sidecar, Codex `config.toml` or OpenCode config gets them. **Reason and benefit** Less code, one path, and no way for a gateway token to reach a provider process through the Runner. ## What Changed - Deleted `packages/paperclip-runner/src/drivers/native-mcp.ts` (`nativeMcpLaunchBinding`). - `drivers/acpx/runtime-host.ts`: removed the native gateway server entry and the name conflict check. The admission gate keeps its message. A run with a binding id is admitted when the sidecar uses the PRP transport with its tool bridge, as the production sidecar always does. - `drivers/acpx/environment.ts`: the name and the URL are not in the provider allowlist. - `drivers/runtime-context-materializer.ts` and `live/runnerd-codex-transport.ts`: the Codex home functions have no `nativeMcp` parameter. The sidecar token pass-through is gone. - `drivers/opencode/opencode-server-driver.ts`: removed the read of the binding, its trace redaction value, and the gateway server entry in the OpenCode config. - `drivers/stdio-mcp.ts` and `runner-core/src/stdio_mcp.rs`: removed the reservation of the gateway server name. No gateway server exists, so no name can conflict. The names `paperclip` and `paperclip_runner_*` stay reserved. - `runner-core/src/acpx_sidecar_transport.rs` and `codex_provider.rs`: the three variables are not in the pass-through lists. - `control-plane/durable-prp-control-plane.ts`: the three variables are not in the platform environment list. - `packages/paperclip-runner/README.md`: the text about direct callers and assigned servers. - Kept: the three names stay in the reserved lists (`src/configured-environment.ts`, `runner-core/src/configured_environment.rs`). A user cannot configure them. Tests in both languages now check all three names. - No server code changed. The relay needs the gateway token and URL on the control plane. 23 files, 73 lines added and 211 lines removed. ## State saved by the previous release - This pull request changes no persisted shape. The provider server lists, the launch profile digests, the protocol versions and the sidecar contract are the same as with #21274. - A Runner or sidecar that the previous release started keeps its own code until its run ends, as #21274 describes. - A kept Codex home that has the old gateway server in `config.toml` is written again at the next start and at a recovery of the same run. A test covers this. - If the old variables reach a new Runner, no provider gets them. Tests cover this for ACPX, Codex and OpenCode. ## Verification From `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/runtime-host.test.ts src/live/runnerd-codex-transport.test.ts src/drivers/acpx/environment.test.ts src/drivers/opencode/opencode-server-driver.test.ts src/live/runnerd-codex-transport-warm-attachment.test.ts src/drivers/runtime-context-materializer.test.ts src/drivers/agent-observability.test.ts src/drivers/stdio-mcp.test.ts src/control-plane/durable-prp-control-plane.test.ts src/configured-environment.test.ts`: `10 passed (10)` files, `807 passed` tests. - `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: no errors. - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no changes. - `cargo test -p paperclip-runner-core --test acpx_sidecar_transport`: 20 passed. `cargo test -p paperclip-runner-core --lib`: 639 passed, 4 ignored. Tests that failed with the old code and pass now: - `runtime-host.test.ts`: "admits a relayed … run with a binding id and ignores a previous-release gateway binding", for Claude, Codex and Cursor. - `runnerd-codex-transport.test.ts`: "drops a previous-release gateway binding from the … runner environment", for ACPX Claude, Codex, Pi, Cursor, Copilot and Kimi, and for OpenCode. - `opencode-server-driver.test.ts`: the OpenCode config has only the task bridge. - `runnerd-codex-transport-warm-attachment.test.ts`: a restored Codex home has no gateway server and no gateway credential in `config.toml`. - `environment.test.ts`: a provider environment has none of the three variables. - Rust `previous_release_gateway_binding_and_unrelated_secrets_never_reach_the_sidecar`: a Runner that starts with the old variables gives none of them to the sidecar, for five agents. Mutation check: with the three names put back in the Rust pass-through list, the Rust test fails. Not run: a live run with a real provider. Not run: `pnpm verify` and the full suite. CI supplies that coverage. ## Risks - **This pull request is safe only after #21274.** The stack order keeps that. - **A developer who sets the three variables by hand for `eval-session`** no longer gets direct gateway tools. No document describes that use. - **The warm attachment test uses the staged Runner binary from preflight.** That binary does not have the Rust changes of this pull request. The Codex path of that test does not use the changed lists. - The evidence that no entry point supplies a binding comes from a read of the code: Core removes the variables before it builds each Runner environment, and a native run cannot use the in-process path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, and ran the verification and a mutation check again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), traced each code path and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
12aa08721c |
feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.
Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on
|
||
|
|
2a8b485140 |
fix(runner): accept object fields sent as JSON text and name wrong-type rejections (#21485)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip runner checks each tool call from the model against the tool's input schema before it sends the call to the control plane > - Some model hosts send an object argument as its JSON text. On the DigitalOcean host, `paperclip_finish` arrived with `continuation` as a string > - The runner rejected these calls, and the repeated-rejection guard then told the model that its arguments "did not arrive", which is wrong when every field arrived with the wrong type > - This pull request makes the runner parse such a string back into its object when that is safe, records each coercion as a run event, and makes the guard name the actual problem > - The benefit is fewer stuck finish calls, and guidance that points the model at the real fault ## Linked Issues or Issue Description Refs #21398 **What happened?** Chat-eval cohort 13 ran OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`. On the DigitalOcean host, 6 of the 8 remaining "Invalid tool input" rejections were `paperclip_finish` calls with `continuation` sent as a JSON string (`/continuation must be object,null`). Examples: `gateways-profiles.rotate-token`, `instance-operations.restart-service`, and `questions-and-approvals.multi-part-question`. The run diagnostic recorded `continuationShape: "string"`. After the second identical rejection, the guard from #21398 said "If you meant to send other arguments, they did not arrive". Every argument had arrived; one had the wrong type. **Expected behavior** When a schema field is object-typed, does not allow text, and the text parses to a plain JSON object that makes the call valid, the runner accepts the call with the parsed object and logs a run event. When a call stays invalid, the guard says which required fields are missing and which fields have the wrong type. **Steps to reproduce** 1. Start the OpenCode MCP bridge. 2. Call `paperclip_finish` with `reportedWorkDisposition: "yielded"` and `continuation` as `JSON.stringify({ kind: "response_wake", ... })`. 3. Before this change, the bridge returns `Invalid tool input: /continuation must be object,null`. ## What Changed - Added `packages/paperclip-runner/src/drivers/tool-input-coercion.ts`. `admitToolInput` validates the arguments. If they are invalid, it parses a string in an object-typed field back into its object. It does this only when the field's schema declares `object`, does not allow `string`, the text parses to a plain JSON object, and the whole coerced call validates. It applies to any object-typed field at any depth, not only `continuation`. - The OpenCode MCP bridge and the runner tool bridge use `admitToolInput`. They dispatch the coerced arguments, and they send one `paperclip.runner.tool-input-coerced/v1` diagnostic per accepted coercion. The OpenCode driver already records bridge diagnostics as run events. The diagnostic names the tool and the field paths. It never contains the values. The runner tool bridge has a new optional `onDiagnostic` option. - When parsed text is still invalid, the rejection describes what is wrong inside the parsed object, for example `/options/mode must be string`. - The repeated-rejection guard now says "Required fields X did not arrive" only for missing required fields. For fields with the wrong type, it says "Every field you sent arrived, but one has the wrong type: continuation must be an object or null, not text". ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/mcp-bridge.test.ts src/drivers/runner-tool-bridge.test.ts` (63 tests pass). - Red/green: with the source changes reverted and the new tests kept, 5 tests fail: coercion acceptance and its run event (both bridges), no coercion for text fields or non-object text, the wrong-type guard wording, and the missing-field guard wording. With the change, all of them pass. - `npx tsc --noEmit -p packages/paperclip-runner` passes. - A narrow paid run of `questions-and-approvals.multi-part-question`, `runs-adapters.inspect-failed-run` and `budgets-costs.explain-hard-stop` ran on a branch that combines this PR with the two sibling cohort-13 PRs. The results are in a PR comment. ## Risks - Low risk. Coercion runs only after a call fails validation, and the call is accepted only if the coerced form validates. A field that allows text is never parsed, so a valid string never changes meaning. - Previous-release story (upgrade-safe run state): this change adds no persisted run, session, launch or pack state. It changes only how one in-flight tool call is admitted. Saved state carries forward unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ebe6d846d |
perf(runner): keep a quiet Claude provider between turns of a run (#21293)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. The Runner stops and starts the sidecar and Claude at each turn boundary. > - The reason was that a late tool call from an earlier turn could run in a later turn. #21278 removes that reason for Claude: the sidecar refuses a call that the active turn did not announce. > - A second reason remains. Claude Code works by itself between prompts. When a background task finishes, Claude starts a cycle of its own. If the process stayed alive with a live background task, that cycle could start between turns or mix with the next turn. > - So the Runner can keep the process only when it knows that Claude has no background work. > - This pull request adds that knowledge and uses it. After a Claude turn that completed with no background work, the Runner keeps the process for the next turn of the same run. > - The benefit is that a later turn of the same run starts with no provider restart. This is also the base for the next step, which lets a run continue when background work finishes. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is stacked on #21278. Its base branch is `fix/bind-claude-tool-calls-to-turn`. Refs #19912 (design), #12419 (restart at turn start), #20145 (restart at park) and #20190 (restart at turn end). **What existing behavior does this improve?** The provider restart between turns of the same run, for Claude through ACPX. **Subsystem affected** Paperclip Runner: the ACPX sidecar (TypeScript) and the ACPX provider session in `runner-core` (Rust). Claude only. **Current behavior** - When a completed turn settles, the Runner restarts the sidecar and Claude (`restart_session_after_turn`, #20190). - If that restart did not run, the next turn restarts them at its start (`start_turn`). - The Runner does not know if Claude has background tasks. **Proposed behavior** - The sidecar asks `claude-agent-acp` for its async task reports. This is an optional capability of the pinned `claude-agent-acp` 0.73.0. The patch does not change. - A tracker reads those reports and the id of each background Bash command. It reports `{ live: 0 }` only when it knows that this process never had a background task and that Claude did not act between prompts. In all other cases the state is unknown. - The sidecar adds `backgroundTasks` to the terminal event of a completed Claude turn, and to `session.snapshot`. - The Runner keeps the process between two turns only when all of these are true: - the agent is Claude; - the turn completed (not failed, cancelled or interrupted); - the terminal event reported `backgroundTasks: { "live": 0 }`; - each existing condition of the turn-end restart is true; - before the next turn, a new `session.snapshot` reports the same identity and run, no active turn, no pending work, and `{ "live": 0 }`; - the settled-turn ledger and the tool replay ledgers are not full. - In each other case the Runner restarts as before. - A kept process keeps its tool call-id tombstones and its settled turn identities. **Reason and benefit** - A turn that follows a quiet turn in the same run starts without a restart. - The decision is safe by default: unknown state means a restart. ## What Changed TypeScript (`packages/paperclip-runner/src`): - `drivers/acpx/claude-background-tasks.ts` (new): `ClaudeBackgroundTaskTracker`. It reads raw ACP messages. It reports unknown when the capability is not negotiated, when a task finished, when a live task is not a shell command, when an update is malformed, when a limit is reached, when Claude acts with no client request in flight, or when Claude calls `CronCreate` or `ScheduleWakeup`. - `drivers/acpx/provider-failure.ts` and `acpx-runtime-adapter.ts`: Claude sessions also advertise the `asyncTasks` capability. The raw ACP observer gives each message to the tracker. ACPX 0.22 drops the `async_task_*` updates before the turn event stream, so only the raw observer sees them. - `drivers/acpx/runtime-host.ts`: `backgroundTaskState()`. - `cli/acpx-runtime-sidecar.ts`: the terminal event of a completed Claude turn carries `backgroundTasks` in the same payload as `goalAtTerminal`, with the same size limit. `session.snapshot` carries it when no turn is active. - `drivers/acpx/claude-tool-use-admission.ts`: `hasRoomForAnotherTurn()`. When the admitted-id ledger cannot hold one more full turn, the sidecar reports unknown, so the Runner restarts before the ledger is full. Rust (`packages/paperclip-runner/runner/crates/runner-core`): - `acpx_provider_session.rs`: the session records a quiet turn from the terminal event. `start_turn` continues in the same process only under the conditions above, and then uses `prepare_turn`, which keeps the tombstones. - `acpx_provider_generation.rs`: `can_continue_after_turn`, `continue_after_quiet_turn`, and the snapshot check `background_work_is_quiet`. - `acpx_park_preparation.rs`: the turn-end restart does not run for a quiet Claude turn. - `acpx_provider_backend.rs`: `session.reconciled` reports a replaced process only when the turn start replaced it. - `bin/fake-acpx-sidecar.rs`: new option `--background-tasks` for tests. - `runner/DURABLE_TRANSPORT.md`: the rule for a kept sidecar. Contract (`packages/paperclip-runner/protocol`): - The `backgroundTasks` report is `$defs.backgroundTasks` in `provider-schemas/acpx-sidecar.schema.json`: an object with only `live`, a non-negative integer. The schema validates it on `runtime.turn_terminal`. - `scripts/generate-acpx-sidecar-contract.mjs` generates the TypeScript type and reader and the Rust reader from it. The tracker, the sidecar and the Rust session use the generated code. - `fixtures/provider-boundary/acpx-background-tasks.json` has shared test documents. The TypeScript reader, the Rust reader and the schema are tested with the same accepted and refused shapes. - This changes the contract digest of the provider pack (`prp.acpx-sidecar` and `rust`). Older image packs are not reused. No provider `commandDigest` and no profile version changes. Not changed: park and run attachment. A process that ran a turn is still restarted at park, and only a process that ran no turn is bound to the next run. Other ACPX agents and Codex do not change. The `claude-agent-acp` patch does not change. ## State saved by the previous release - This pull request adds no persisted state. The three new fields of the session are in memory only. - A terminal event or a snapshot without `backgroundTasks` means unknown. An older sidecar therefore gets the restart, as before. - An older Runner with a new pack ignores the field and restarts at each turn, as before. - After a Runner restart, the in-memory state is gone. The next turn restarts the provider, as before. A test covers this. ## Verification From `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/claude-background-tasks.test.ts src/drivers/acpx/acpx-runtime-adapter.claude-background-tasks.test.ts src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts src/drivers/acpx/claude-tool-use-admission.test.ts src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx/acpx-runtime-adapter.test.ts src/drivers/acpx/runtime-host.test.ts`: `7 passed (7)` files, `363 passed` tests. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json`: no errors. - `pnpm run typecheck:typescript` (it includes the contract checks): no errors. `node --test test/acpx-sidecar-contract.test.mjs`: 17 passed. - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no changes. - `cargo test -p paperclip-runner-core --test acpx_quiet_turn_continuation --test acpx_park_preparation --test acpx_park_readiness --test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment --test native_provider_backend`: 4, 33, 8, 31, 12, 6 and 47 passed. No test failed. - `cargo test -p paperclip-runner-core --lib`: 637 passed, 4 ignored. - Mutation check: when the Rust code accepts each live count as quiet, two tests in `acpx_park_preparation` fail. New tests: - **Real ACP runtime** (`acpx-runtime-adapter.claude-background-tasks.test.ts`), with the pinned `claude-agent-acp`, the pinned Claude Code client and a local fake Anthropic API. A foreground Bash turn reports `{ live: 0 }`. A background Bash in the same process reports `{ live: 1 }` at the terminal event, and unknown after Claude ran its own cycle for the finished task. Each turn has its own usage receipt. One process serves all turns. - **Tracker unit tests** (13): spawn, completion, failure, stop, pause, a task that is not a shell, a subagent, no negotiation, malformed updates, activity between prompts, scheduling tools, session load replay, limits. - **Sidecar tests**: the terminal event carries the state for Claude and not for Codex. A full ledger reports unknown. - **Rust** `tests/acpx_quiet_turn_continuation.rs` (4) and `tests/support/acpx_quiet_turn_restart.rs` (7): - a quiet Claude turn continues in the same process, and a replay of a settled call id is refused; - live, unknown and malformed reports restart; - background work that appears after the terminal event restarts at turn start; - failed, cancelled and interrupted turns restart; - Codex with a quiet report restarts; - an active goal restarts; - 4,096 kept turns, then the next turn restarts; - park after a kept process still prepares a new process, and the next run attaches to it; - a Runner restart after a kept process resumes in a new process. - The existing restart tests did not change. The fake sidecar sends no report by default, so they are the tests for the unknown case. Live run with a real Claude account, not in CI: the real ACP runtime (ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code client), a real tool bridge, and the real Anthropic API with model `claude-sonnet-5-5`, on the code at the top of this stack. A turn that started one background shell reported `{ live: 1 }`. A turn that started two reported `{ live: 2 }`, then `{ live: 1 }` and `{ live: 0 }` as each shell finished. A turn in which Claude waited for its background shell in the foreground reported the unknown state. Not run: server integration tests (the fake sidecar there sends no report, so their process counts do not change), Linux, and a live run through the real sidecar process and runnerd. ## Risks - **All Claude sessions now negotiate the `asyncTasks` capability.** `claude-agent-acp` then sends `async_task_*` updates and adds a marker to a background Bash update. ACPX drops the updates. No other behavior of `claude-agent-acp` changes, from a read of its code. - **Park after a kept process pays the restart again**, as it did before #20190. With real Claude that restart was about 1.5 s. - **A small window remains** between the snapshot and the prompt of the next turn. Claude can start a cycle of its own in that window only without a background task, because a process that had a background task is not kept. - **Activity between prompts is found only when no client request is in flight.** A request that never gets a response would hide such activity. - **Background watchers cannot be told apart from other tasks**, so each task that is not a shell command makes the state unknown. - **Each kept turn adds one `session.snapshot` request.** - A multi-turn run with real Claude now uses one provider process. This path has tests with the fake sidecar and with the real ACP runtime. It has no test with the real sidecar host. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, and ran the TypeScript and Rust verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), investigated the signals and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
7773b52f05 |
fix(codex): preserve sandbox certificate trust in Codex shell commands (#20364)
<!-- Written in ASD-STE100 style: short sentences, active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner and the Codex adapters run agent shell commands inside a sandboxed Codex session with a strict environment allowlist > - The controller builds `shell_environment_policy.include_only` before the sandbox injects certificate-trust variables such as `SSL_CERT_FILE` and `GIT_SSL_CAINFO` > - The allowlist drops the injected variables, so managed `git` and `gh` commands cannot verify HTTPS against a runtime-provided CA > - This pull request allows the certificate-trust variable names through every environment policy stage, from the Rust process supervisor to the shell allowlist > - The benefit is that managed `git` and `gh` work again in sandboxes that require a runtime CA, while provider-credential isolation and the bounded allowlist stay in place ## Linked Issues or Issue Description - Fixes: #20019 - Refs #20038 — community pull request for the same defect. This change reviews the approach independently. The commit credits the community author as co-author. ## What Changed - Add `CERTIFICATE_ENVIRONMENT_KEYS` (`SSL_CERT_FILE`, `SSL_CERT_DIR`, `GIT_SSL_CAINFO`, `NODE_EXTRA_CA_CERTS`, `DENO_CERT`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`) to `packages/paperclip-runner/src/configured-environment.ts` - Native Codex shell policy (`codex-security-config.ts`): always put the certificate names in `include_only`; set inheritance to `"all"` with `ignore_default_excludes=true`, bounded by the explicit allowlist - Pass the certificate names through the app-server environment filter, the ACPX spawn environment filter, the ACPX runtime sandbox `include_only`, the runner platform environment keys, and the Rust process supervisor base key list - Legacy and ACPX-engine Codex shell allowlists (`packages/adapter-utils/src/codex-shell-environment.ts`): always include the certificate names - Add a synthetic private-CA HTTPS journey test: a real `git ls-remote` fails TLS verification without the injected CA and succeeds with it, through the policy-filtered child environment - Document the contract in `doc/sandbox-shell-environment.md` and link it from `doc/DEVELOPING.md` ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/codex/codex-security-config.test.ts src/configured-environment.test.ts src/drivers/codex/codex-shell-certificate-trust.test.ts src/drivers/codex/codex-app-server-driver.context-control.test.ts src/drivers/acpx/runtime-sandbox.test.ts` — 104 tests pass - `pnpm --filter @paperclipai/adapter-utils exec vitest run src/codex-shell-environment.test.ts src/acpx-engine/execute.test.ts` — 270 tests pass - `pnpm --filter @paperclipai/adapter-codex-local exec vitest run src/server/execute.test.ts` — 10 tests pass - Codex driver lifecycle, conformance, recovery, and boundaries suites pass (103 tests); environment-builder suites pass (307 tests) - Red proof: with only the source fix reverted, the new regressions fail. The journey test shows `GIT_SSL_CAINFO` is undefined in the shell environment, and `git` fails TLS verification against the private-CA fixture - `cargo check` passes for `paperclip-runner-core`; `rustfmt --check` passes for the edited Rust file - The journey test needs `git` and `openssl`; it skips when either tool is absent ## Risks - The native shell policy now always sets `shell_environment_policy.inherit="all"` with `ignore_default_excludes=true`. Before, it set `"none"` when no credentials or task keys were projected. The explicit `include_only` allowlist still bounds every inherited name, so no new secret names can cross - The allowlist grows by seven non-secret certificate-trust names. Their values point at CA files. The sandbox filesystem policy still controls access to those files; this change grants no filesystem access - No persisted run, session, or launch identity changes shape. Sessions saved by the previous release resume unchanged; the new policy applies when the next launch builds its arguments - Provider credentials keep their existing exclusions at every stage ## Model Used - Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended thinking, agentic tool use, run through Claude Code. The implementation was written and reviewed independently against the community design in #20038; the community author is credited as co-author in the commit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending: checks run after this PR opens) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending review) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: everlastingcrown <everlastingcrown@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
7866e9ef05 |
fix(acpx): admit a Claude tool call only for a tool use its turn announced (#21278)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. Claude calls Paperclip tools through a Runner-owned MCP endpoint, the tool bridge. > - The bridge identifies a call by its JSON-RPC request id. That id does not say which turn sent the call. The sidecar binds a call to the turn that is active when the call arrives. > - Because of this, the Runner stops and starts the Claude process at each turn boundary. A late call from an earlier turn then cannot run in a later turn. This restart also stops all background work that Claude started. > - Claude Code already sends the missing identity. Each MCP `tools/call` has `_meta["claudecode/toolUseId"]`. It is the id of the ACP `tool_call` that the provider announces first, inside one turn. > - This pull request uses that id. The sidecar admits a Claude tool call only if the active turn announced its tool use. > - The benefit is that a call from an earlier turn is refused by its identity. This is the first step to keep the Claude process alive between turns. This pull request does not change the restart. ## Linked Issues or Issue Description No public issue exists. The issue is described here. Refs #19912 (the design for turn-scoped ACPX MCP authority), #12419 (the restart at turn start) and #20190 (the restart at turn end). This pull request is the bottom of a stack. #21274 is merged, so the base branch is `master`. It uses no code from #21274. The stack order lets a later pull request remove the direct gateway path after both. **What existing behavior does this improve?** How the Runner decides which turn a Claude tool call belongs to. **Subsystem affected** Paperclip Runner: the tool bridge and the ACPX sidecar, for the Claude provider only. **Current behavior** - The bridge keys a call by the JSON-RPC request id (`runner-tool-bridge.ts`). - The sidecar gives the call to the active turn (`waitForTool` in `acpx-runtime-sidecar.ts`). - After `session/cancel`, Claude Code does not stop an MCP request that is in flight. A test run showed two calls from two turns live at the same time on one MCP session. Request ids cannot tell them apart. **Proposed behavior** - The bridge can read a provider tool-use id from `params._meta` and give it to the handler. It keys replays by that id. - For Claude, the sidecar admits a call only when the turn event stream of the active turn contains the ACP `tool_call` with the same id, and no earlier call used that id. - The announcement and the call use different channels. If the call arrives first, the sidecar waits for the announcement for a maximum of 5 seconds. The wait stops when the turn ends or the call is cancelled. - A call with no id, an id from an earlier turn, an id that was never announced, or an id that was used before gets a tool error. There is no fallback. - The call id that the sidecar sends to runnerd is `tool-use:<id>` for Claude. **Reason and benefit** - A late call from an earlier turn cannot run in a later turn. The identity of the call prevents this, not only the process restart. - A later change can stop the restart between turns. ## What Changed - `packages/paperclip-runner/src/drivers/runner-tool-bridge.ts` - New option `providerToolUseIdMetaKey`. Without it the bridge does not change. - With it, the bridge reads `params._meta[key]`. It refuses a value that is not a bounded string. It gives a valid value to the handler as `providerToolUseId`. - The replay key is the tool-use id. A retry with a new request id replays the first result. The same id with different arguments is refused. - `notifications/cancelled` still finds the call by its request id. - `packages/paperclip-runner/src/drivers/acpx/claude-tool-use-admission.ts` (new): `ClaudeToolUseAdmission`. - `observe` records an id only from an event with `type === "tool_call"` and `tag === "tool_call"`. A `tool_call_update` does not count, so a late update about an old tool use cannot admit it. - `admit` applies the rule above. An id is used when it is admitted. Limits: 4,096 announcements for each turn and 8,192 admitted ids. At a limit, new calls are refused. - `packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts` - The sidecar creates the admission object only for `agent === "claude"` and sets the bridge option only for Claude. - `pumpTurn` gives each turn event to `observe`. `waitForTool` calls `admit` and then checks the turn and run again. - The internal `get_task_context` call for a completion continuation does not go through admission. - The Claude call id does not use the per-process namespace, because a tool-use id is unique. - No change to Rust, the sidecar contract, the `claude-agent-acp` patch, the server, or restart, park and turn lifecycle code. Other ACPX agents and Cursor Cloud use the same code path as before. Why 5 seconds: the measured time between the announcement and the call is 1.1 ms to 14.7 ms. The limit must also include queue and event-loop delay. 5 seconds is far above that, and a call that cannot be attributed still gets a fast answer. ## Verification Commands, from `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts src/drivers/acpx/claude-tool-use-admission.test.ts src/drivers/runner-tool-bridge.test.ts src/drivers/runner-tool-bridge.claude-tool-use.test.ts src/drivers/runner-tool-bridge.claude-schema.test.ts src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx/acpx-runtime-adapter.test.ts src/drivers/cursor-cloud/cursor-cloud-driver.test.ts`: `8 passed (8)` files, `384 passed` tests. - `npx tsc --noEmit -p tsconfig.json`: no errors. New tests: - **Real ACP runtime** (`acpx-runtime-adapter.claude-tool-use.test.ts`). It uses the real `openCodexAcpxRuntime`, ACPX 0.22, the pinned and patched `claude-agent-acp`, the pinned Claude Code client and a real tool bridge, against a local fake Anthropic API. - Turn 1: two parallel tool uses are admitted as `tool-use:<id>`. The model receives both results. The turn completes. - Turn 2 in the same session: a replay of a turn 1 id returns the cached result and does not dispatch again. The same id with different arguments is refused. `admit` refuses the old id. - In five runs each call arrived 6.9 ms to 13.2 ms after its announcement. No call waited. - **Qualification** (`runner-tool-bridge.claude-tool-use.test.ts`). The pinned Claude Code client sends `_meta["claudecode/toolUseId"]` on each `tools/call`, with the id that the API issued. This test must pass before a Claude Code version change. - **Admission unit tests** (19): announced then called, called then announced, never announced, turn end during the wait, cancel, an id from the previous turn, reuse, two concurrent calls with one id, two turns with live calls, a subagent call, a bad id. - **Sidecar process tests** (2), with a fake host, for Claude and for Codex. The Codex run shows no change. - **Bridge unit tests** (5). Mutation checks: - Change the announcement predicate so that it matches nothing: the real ACP test and one unit test fail. - Remove the bridge option in the real ACP test: the test fails. - Count only `tool_call_update`: the unit test "counts only an opening ACP tool_call event as an announcement" fails. Live run with a real Claude account, not in CI: the real ACP runtime (ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code client), a real tool bridge, and the real Anthropic API with model `claude-sonnet-5-5`, on the code at the top of this stack. In three sessions, 8 Paperclip tool calls were admitted and none was refused. This includes calls in a prompt turn, in a cycle that Claude started itself, and in a later turn of the same process. Each call arrived 0.8 s to 1.7 s after its announcement, so no call waited for a late announcement. Not run: a turn through the real sidecar process with the real host. That needs a qualified provider installation, the runtime sandbox and the provider lifetime lease. Not run: the Linux builds of the pinned client and a test under load. Not run: `pnpm verify` and the full suite. CI supplies that coverage. ## Risks - **This changes each Paperclip tool call that Claude makes.** If the announcement does not reach the turn event stream in production, the sidecar refuses the call after 5 seconds. The real ACP test covers the stream of ACPX 0.22 with the pinned provider. It does not cover the real sidecar host. - **The 5 second limit is not tested under load.** A slow host can delay the announcement. The call is then refused, and Claude sees a tool error. - **A Claude Code version that stops sending the id breaks all Claude tool calls.** The qualification test finds this before the version changes. - **The admitted-id ledger is in sidecar memory.** Today the sidecar restarts at each turn boundary, so the ledger is small. The limit of 8,192 ids refuses new calls when it is full. - **Subagent calls and calls from a cycle that Claude starts itself** are covered by unit tests and fake-host tests, not by the real ACP stream. - **An admitted call that is in flight when its turn ends** is refused by the existing turn cleanup. Claude Code can leave the HTTP request open. The bridge stops it only on its own cancel or close, as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, asked for the real ACP runtime test, and ran the verification and a mutation check again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
38821e061f |
perf(runner): serialise only changed journal records for a checkpoint (#21437)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every running agent, and its main thread limits how many agents can run at the same time > - Each agent has a controller journal. The controller saves a checkpoint of this journal after each control event and before each frame it sends > - Each checkpoint converted every command and every event of the journal to JSON again, also when only one record changed. The cost grew with the journal length (up to 500 commands and 4,096 events) > - This pull request keeps the JSON of each record and uses it again while the record does not change > - The benefit is that a checkpoint converts only the records that changed, so the main thread does less work for each agent ## Linked Issues or Issue Description Refs #21411 (this change sits on top of its checkpoint queue). **What existing behavior does this improve?** The controller journal checkpoint in `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`. Each flush calls `captureCheckpoint`, which runs `JSON.stringify` on every command and every recovery event of the journal. **Current behavior** A flush after one new event converts the full journal to JSON. With 4,096 events, one flush made about 5 MB of new JSON strings and used about 14 ms of CPU on the event loop. A hosted CPU profile with about 50 agents showed about 3% of the main thread in this path. **Proposed behavior** A flush converts only the records that changed since the last flush. The other records use the JSON that the controller made before. The saved records are byte-identical to the records of the full conversion. **Reason and benefit** The target is 1,000 agents in one controller process. Checkpoint CPU must not grow with journal length. After this change, one flush with 4,096 events makes about 2.5 KB of new JSON and uses about 2 ms of CPU. ## What Changed - Add `CheckpointRecordCache` to `durable-prp-control-plane.ts`. It keeps the JSON and the part name of each command and event record, keyed by the record object. It uses the JSON again only when the record has the same keys, in the same order, with the same values (`===`). Commands change some fields in place (`status`, `result`, `controllerSeq`), and this comparison finds these changes. - `captureCheckpoint` accepts the cache as an optional fourth argument. Only `setCheckpointPersistence` and `flushCheckpoint` pass it. All other callers (decode, legacy import, bootstrap inspection) do the full conversion as before. - `checkpointChanges` uses loops instead of array copies. Records that did not change keep the same string, so the comparison is by reference. - `queueCommand` stores a copy of the command payload (`structuredClone`). The journal must own its nested objects, because the cache compares nested objects by reference. - `durableRecoveryInternals` exposes `captureCheckpoint` and `createCheckpointRecordCache` for tests. - Regions changed: `checkpointChanges` and the new cache above `captureCheckpoint`; `captureCheckpoint`; `durableRecoveryInternals`; one field of `DurablePrpControlPlane`; the `captureCheckpoint` calls in `setCheckpointPersistence` and `flushCheckpoint`; the `payload` field in `queueCommand`. The `#event` path is not changed. - No change to what a checkpoint saves, when a flush is awaited, or the order of event commit, acknowledgement and checkpoint. The "at most one waiting checkpoint" behavior from #21411 is not changed. ## Verification - New `durable-prp-checkpoint-records.test.ts`: - A journal with 499 commands and 4,000 events gets one new command. The test counts `JSON.stringify` calls on journal records during the flush. Before: 4,499. After: 0 calls on existing records, at most 2 in total (the size check of the new command and its checkpoint record). The delta contains only `command:500`. The test then restores a new controller from the checkpoint and compares its records with a full conversion without the cache. I saw this test fail on `origin/master`. - A property-style test with 5 seeds and 600 random steps each: add, settle and remove commands (with `controllerSeq` renumbering), add, replay and evict events, streaming and usage events, `structuredClone` and spread copies of the state, header counters, the same value written again, and a key-order change. After each step, the records with the cache must be byte-identical to the records without the cache, in the same order. I checked that this test fails when the field comparison is removed (cache by object only). - Existing `durable-prp-control-plane.test.ts` passes. On this loaded machine, some 5-second timing tests time out with and without this change (they also time out on `origin/master`), and pass when I run them alone. - Benchmark: `PAPERCLIP_CHECKPOINT_FLUSH_BENCHMARK=1 pnpm exec vitest run src/control-plane/durable-prp-checkpoint.benchmark.test.ts` (from `packages/paperclip-runner`). Each case commits one new event and flushes with a no-op save, 200 times. CPU time is noisy (load average about 40), the byte and record counts are exact. | Journal | CPU per flush, before | CPU per flush, after | JSON bytes per flush, before | after | Records converted per flush, before | after | | --- | --- | --- | --- | --- | --- | --- | | 10 commands, 10 events | 207 µs | 64–80 µs | 20,975 | 2,465 | 20 | 1 | | 500 commands, 10 events | 1,874 µs | 505–511 µs | 375,068 | 2,466 | 510 | 1 | | 10 commands, 4,096 events | 14,013 µs | 1,655–2,343 µs | 5,174,786 | 2,473 | 4,106 | 1 | - `pnpm check:module-boundaries` passes. `tsc -p packages/paperclip-runner/tsconfig.json --noEmit` passes. ## Risks - The cache assumes that nested `payload`, `result` and `envelope` objects of journal records are not edited in place after they enter the journal. I checked the module: results are stored with `structuredClone`, envelopes with `structuredClone`, and events are replaced, not edited. This change also copies the payload in `queueCommand`. If a later change edits a nested object in place, the checkpoint could keep old JSON. The property test shows top-level edits are found. - Some per-record work is still proportional to the journal (a Map of part names and a field comparison for each record). It is about 0.4 µs for each record. Removing it needs dirty tracking in every state mutator, which this change does not do. - No change to the saved format, so old and new servers read the same checkpoints during a deploy overlap. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d9b977baf2 |
fix(runner): answer a live Runner's commands while it replays its backlog after a reconnect (#21442)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox is driven by a Runner process. The
server talks to it over PRP and saves the controller state in a
checkpoint.
> - After a release swap, the incoming server reconnects to Runners that
kept working while no server was attached. Each Runner first replays the
events it built up.
> - The server answered commands one at a time and only after every
event queued ahead of them. It wrote one checkpoint and one
acknowledgement per event. A recovery read sent at reconnect therefore
timed out at 30 s behind the replay, although the Runner was alive and
had answered.
> - A tool call that the previous server started and never answered
blocked every later command, and a failed close was replayed forever.
> - This pull request lets read and barrier answers pass the replay,
extends a wait while the replay still commits, acknowledges a replayed
burst once per batch, delivers every pending command at attach, settles
an orphaned tool call at attach, and retries a close whose drain timed
out.
> - The benefit is that a live Runner is reached after a late reconnect
instead of being declared unanswered and then replaced, which lost its
turn.
## Linked Issues or Issue Description
**What happened?**
Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:
- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed
Authentication took under 1 s. The identity was recorded 4 to 7 minutes
later, because it is written only after the Runner answers `run.attach`,
`runner.drain` and the thread read. Those answers waited behind the
replayed event backlog. The failure codes on the first attempt were:
- `PRP command session.snapshot timed out` and
`codex_history_read_failed: thread/turns/list read failed` (all 10
`ingress` runs in the sample)
- `PRP command runner.drain timed out (runner connected; pending ahead 2
[semantic_tool.result] oldest 1023s)` on the retry
A separate class, `cleanup_quarantined` (23 failures in 3 hours, no swap
involved), showed "prior session cleanup remains incomplete" on every
retry, because a close that failed once kept replaying the same
rejection.
A live Daytona capture showed the stale `semantic_tool.result`: Codex
waited for the result of an exec tool call that the stopped server
owned. It was not an unacknowledged frame.
**Expected behavior**
A live Runner answers recovery commands within its deadline after a late
reconnect. A tool call left by an earlier controller is settled once
(with its durable receipt, or as interrupted with an unknown outcome,
never executed again). A close that failed only because its journal
drain timed out reaches its real outcome on the next close.
**Steps to reproduce**
1. Start a control plane with a commit callback that takes 50 ms per
event.
2. Connect a Runner that replays 1,000 events, then send
`session.snapshot`.
3. On master the snapshot times out at its 30 s deadline. The new test
`answers a command issued during a 1,000-event replay within its
deadline` does this.
**Paperclip version or commit**
`master` at
|