mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
dfd2a6efcab84bc19106a2dbf720eb53cb2282d3
111
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
a52f622be8 |
fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error
## Linked Issues or Issue Description
Refs #20808
Refs #20832
Refs #20820
**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.
**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
4529dfb77f |
fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only
## Linked Issues or Issue Description
Refs #20808
**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.
**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.
**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.
## What Changed
- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.
## Verification
- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in
|
||
|
Nicky LeachandClaude Opus 5.5
|
ff8ec41e6a |
feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude
|
94c7c83ce8 |
fix(runner): refuse a moved-root same-session checkpoint by name (#20723)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox resume a task's provider session from a saved checkpoint > - #20712 made a checkpoint that names an abandoned remote workspace root decline recovery, so a replacement session opens in the new root > - A checkpoint that requires the same session (an unfinished goal, `same_session_only`) cannot be replaced. It then failed with the generic `native_session_recovery_failed`, which recovery treats as a retryable interruption > - A retry cannot move a saved session to another root, so the failure repeats. The upgrade-safe rule requires a permanent, named code for this case > - Also, the working-directory errors said "Codex" on Claude and other ACPX runs, because one app-server driver serves all of them > - This pull request reports the moved root as provider state that is incompatible, and makes the errors name the agent generically > - The benefit is a named, permanent refusal with a working board Retry, and error text that is correct for every agent ## Linked Issues or Issue Description Refs #20712 **What happened?** After #20712, a `same_session_only` checkpoint with a moved remote root ended with `native_session_recovery_failed: persisted session working directory is not the assigned remote workspace`. That is not a named permanent code, so recovery could retry a run that cannot succeed. On ACPX Claude runs, path errors read "Remote Codex working directory ...". **Expected behavior** The run ends with the permanent code `native_provider_state_incompatible`. Automatic retries stop. A board Retry continues the task in a fresh provider session in the same workspace. Errors do not name Codex on other agents. **Steps to reproduce** 1. Save a native checkpoint with `providerRecoveryPolicy: same_session_only` under one remote workspace root. 2. Assign the run a different sibling root (the epoch was abandoned). 3. Resume the run. ## What Changed - `codex-app-server-driver-impl.ts`: the moved-root decline from #20712 now carries `failureClass: provider_state_incompatible`. The runtime already maps that class: - checkpoint allows replacement (`allow_replacement_after_resume_failure`): a replacement session opens in the assigned root, as before; - checkpoint requires the same session: `NativeProviderStateIncompatibleError`, code `native_provider_state_incompatible`. - The server already treats `native_provider_state_incompatible` as a permanent failure (no automatic retry, board action text "Retry the task to continue in a fresh provider session with the same workspace"). `rebindNativeSessionCheckpoint` returns no checkpoint after that code, so a board Retry starts a fresh session in the same workspace. No server change is needed. - A moved-root checkpoint with an active turn or a pending runtime request now reports `provider_side_effect_pending`. The runtime never replaces it under any policy, and the run ends with the permanent code `native_provider_side_effect_pending` for an operator to reconcile. Before, it threw the working-directory refusal, which recovery recorded as the retryable `native_session_interrupted`. - `harness-driver.ts`: the `PROVIDER_STATE_INCOMPATIBLE` doc now names the moved-workspace case. - `codex-boundaries.ts`: working-directory errors say "Agent working directory", "Remote agent working directory" and "Provider thread response" instead of "Codex". Tests updated to the new text. - Tests: - `native-session-runtime.test.ts`: the real app-server backend with `remote_runner` authority and a moved root, for each recovery policy. Same-session policies end with `native_provider_state_incompatible`, the replacement policy opens a replacement in the assigned root, an in-flight turn ends with `native_provider_side_effect_pending` under every policy, and no transport starts. - `native-backend-factory.test.ts`: the decline carries the failure class, and an ACPX Claude refusal reads "Remote agent working directory does not match the assigned workspace". Previous-release story: no persisted shape changes. Saved checkpoints are classified at recovery time. A saved same-session checkpoint under an old root now ends with the named permanent code instead of a generic one. ## Verification - Red: without the failure classes, the same-session and in-flight runtime cases fail (generic `native_session_recovery_failed`). Before the rename, the ACPX Claude message test fails. - Green: `cd packages/paperclip-runner && npx vitest run src/backends src/drivers/codex src/live/runner-prepared-launch-workspace.test.ts src/native-session-runtime.test.ts` passes (784 tests). - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Existing server coverage for the code: `native-session-resume.test.ts` (no rebind after `native_provider_state_incompatible`) and `native-session-executor.test.ts` (source-code mapping). ## Risks - Low. Error message text changes. A search found no code that matches on the old text outside the updated tests. - Same-session checkpoints with a moved root now stop for a board Retry instead of a generic interruption. That is the intended behavior: a retry cannot succeed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
1fa7e92fa8 |
fix(native): restart in place a Runner that only settled an earlier attempt's suspend (#20724)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run talks to a Runner process. When the control plane loses
that Runner, the run resumes itself with a limited number of same-run
attempts.
> - A server freeze while the controller suspends a finished turn can
stop the Runner inside `runner.suspend`. The controller journal then
keeps that suspend as pending.
> - The next attempt starts a new Runner. It settles the old suspend
from the welcome, records `suspended`, and exits with code 0 before it
accepts any command of the new attempt. The server reported "runnerd
exited unexpectedly with code 0", so one freeze spent one same-run
attempt. If a second fault then occurs, no attempt is left in reserve.
> - #20714 made both sides agree on that suspend's acknowledgement. This
pull request handles the clean exit that follows it.
> - During startup, the server restarts that Runner in place once,
through the existing `#recoverRunnerProcess` path, and only when the
durable Runner journal proves the exit.
> - The benefit is that one freeze during a suspend costs zero same-run
attempts.
## Linked Issues or Issue Description
Refs #20714.
**What happened?**
A restart-chaos task (local Runner, Claude, a freeze of the server
during a live turn) needed a manual Retry. #20714 fixed the identity
mismatch. The second same-run attempt still ended with
`runner_local_connect_failed: runnerd exited unexpectedly with code 0`,
because that Runner only settled the suspend left by the first attempt
and then exited as suspended.
**Expected behavior**
The attempt continues. The server restarts the Runner once. The new
Runner starts from a clean `suspended` state and serves the attempt's
commands. No provider work is repeated.
**Steps to reproduce**
1. Start a native run. Let the controller send `runner.suspend` when it
closes the session.
2. Stop the Runner after it journaled the suspend and before the
controller records the result (a server freeze does this).
3. Resume the same run. The new Runner settles the old suspend and exits
with code 0, and startup fails.
The new live test reproduces this with the real runnerd and the fake
Codex provider.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
e719d42557 |
fix(runner): accept the acknowledgement of an indeterminate suspend replay (#20714)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run talks to a Runner process (runnerd). The controller in the server sends commands. The Runner journals each command before it acts and replies with a result. > - `runner.suspend` and `runner.shutdown` are terminal commands. The Runner waits for a `command_result_ack` before it exits. It compares that acknowledgement with the receipt that it journaled. > - A Runner that is killed inside `runner.suspend` journals the command but not its result. The next Runner replays the command as `indeterminate`. It sends it on the wire as `failed` with `execution_indeterminate`, and it still treats it as terminal. > - The controller acknowledged the `failed` status that it received. The Runner compared that with its journal status `indeterminate` and exited with "terminal command result acknowledgement changed its durable identity". The controller also lifted its terminal fence for a failed suspend and sent the next command to a Runner that was about to exit. > - This pull request makes both sides agree on the status of that one receipt, and keeps the fence. > - The benefit is that a freeze or restart during a suspend no longer burns the same-run retries and leaves the task at a manual Retry. ## Linked Issues or Issue Description **What happened?** A restart-chaos task (local Runner, Claude, a 28-second freeze of the server during a live turn) needed a manual Retry. Retry worked. 1. The turn completed. The server then sent `runner.suspend`. The server froze before the Runner reported the result. 2. The close found the Runner not durably suspended (`commandStatus: pending`, `runnerLifecycle: ready`). The run failed with `native_runner_checkpoint_owner_lost`, and same-run recovery stopped the old runnerd. The suspend was journaled but not finished. 3. Attempt 2 started a new runnerd on the same state. The welcome delivered the stale suspend. The Runner replayed it as `indeterminate` and sent `failed`. The controller acknowledged `failed`. The Runner exited with code 1: "terminal command result acknowledgement changed its durable identity". The controller had recorded the suspend as `failed`. 4. Attempt 3 found the fence for a suspend that the controller no longer listed as pending. The Runner reconciled it and exited with code 0. The server reported "runnerd exited unexpectedly with code 0", and `native_session_retry_exhausted` stopped automatic recovery. **Expected behavior** The controller acknowledges exactly the receipt that the Runner sent. The Runner accepts that acknowledgement, records `suspended`, and the next attempt starts from a clean suspended Runner. **Steps to reproduce** 1. Start a native run with a local Runner. Let the turn complete so that the server sends `runner.suspend`. 2. Freeze the server (SIGSTOP) for about 30 seconds before the suspend result arrives, so that recovery stops the Runner inside the suspend. 3. Let same-run recovery start a new Runner. It exits with "terminal command result acknowledgement changed its durable identity". The unit tests in this PR reproduce the same sequence without a live run. **Paperclip version or commit** `master` at |
||
|
DottaandClaude
|
3fdcc2a28c |
fix(runner): resend an ACPX turn whose start was cut off before the prompt left (#20675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) sends `turn.start` to its Runner. The Runner asks the ACPX sidecar to start the turn, and the sidecar sends the prompt to the provider > - A start can fail without a definite answer. A host freeze can outlast the sidecar request deadline, or the sidecar or Runner can die during the start > - The Runner then saved the session `closed`. Every same-run resume failed with `ACPX session is not open`, and the task needed Retry > - The Runner cannot tell from its own state whether the provider has the prompt. Resending could duplicate it. A continuation could lose it > - This pull request adds a durable record that answers that question: a sidecar prompt-dispatch journal, written before the prompt reaches the provider > - The benefit is that a start cut off before the prompt left resends the prompt exactly once, with no Retry. Every case the record cannot prove keeps today's behavior ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude, workload C1, fault TM2). A follow-up run reopened the parked Claude session and sent `turn.start`. The sidecar first starts Claude and loads the conversation; the prompt goes out only after that. After 24 s without a reply, the harness froze the server, Runner, sidecar and Claude (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar deadline had passed. It reaped the sidecar and saved the session `closed`. Three same-run resumes failed with `PRP command turn.start failed: … ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and a person had to click Retry. **Expected behavior** When a durable record proves the prompt never reached the provider, the run resends it once by itself. When the record cannot prove it, the run behaves as today. **Steps to reproduce** 1. On master, run `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_park_preparation cut_turn_start`. 2. `a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once` fails: the session is `closed` and the next start fails with `ACPX session is not open`. Refs #20507, #20645, #20674 ## What Changed - New file contract in `protocol/provider-schemas/acpx-sidecar.schema.json`: `turnDispatchJournalFile` (`paperclip-turn-dispatch.jsonl`) and `turnDispatchJournalRecord` (`received` / `dispatched`). The generator emits the file name and schema into TypeScript and Rust, so the pack contract digest covers them. - Sidecar (`turn-dispatch-journal.ts`, `codex-runtime-adapter.ts`, `runtime-host.ts`, `acpx-runtime-sidecar.ts`): when the runtime port starts a turn, before any provider work, it truncates the journal and writes `received`. ACPX calls `onAcpMessage("outbound")` synchronously before it writes a message to the provider. On the turn's first `session/prompt` request the port appends `dispatched` and fsyncs it there, so the record exists before the bytes leave. A failed dispatch write aborts the prompt write. A failed `received` write removes the file and runs the turn unjournaled. - Runner (`acpx_turn_dispatch.rs`, `acpx_provider_backend.rs`): the journal is read only after the provider lifetime fence proves that the sidecar and provider are gone. A lone complete `received` record for the turn proves the prompt never left. - A start that failed in flight saves the session `prepared` with no active turn and returns `ACPX turn start failed before its prompt reached the provider`. The run's next `turn.start` reopens the conversation and sends the prompt once. Until the proof settles, the saved state stays `turn_starting`, so a Runner killed during the proof goes through the restore check below. - A Runner killed while the session was `turn_starting` restores the session idle (`session.resumed`, no `turn.failed`). The same-run resume then starts the turn once instead of sending a continuation that would lose the prompt. - A `dispatched` record, another turn's record, a torn or malformed file, or no file keeps today's behavior (closed session, or lost-turn recovery). - Fake sidecar: writes the same journal and adds `hang-before-dispatch` / `hang-after-dispatch` faults. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` ("ACPX turn starts cut off before the prompt left"). The design note is `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md` (in #20674). ## Verification - Red on master, green here: `cargo test -p paperclip-runner-core --test acpx_park_preparation cut_turn_start` (4 tests: cut before dispatch reopens and sends once; cut after dispatch stays closed and never resends; Runner killed before dispatch restores idle; Runner killed after dispatch or with no journal reports the lost turn). The two "before" cases fail on master. - `cargo test -p paperclip-runner-core`: all pass. `cargo fmt --check` clean. - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx/turn-dispatch-journal.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts`: 121 passed. One test uses a real ACPX runtime and an ACP peer that refuses a prompt arriving before its `dispatched` record; it fails if the hook is removed. - `node scripts/generate-acpx-sidecar-contract.mjs --check`, `node --test scripts/contracts/contract-fingerprints.test.mjs`, and `tsc -p tsconfig.test.json` pass. - Time to first token, measured with the same open/write/fsync sequence on this Mac (APFS, load average ~11, 300 samples): `received` p50 0.24 ms, p95 0.59 ms; `dispatched` (with fsync) p50 4.7 ms, p95 11.5 ms. Against a Claude TTFT budget of 9.5–16.5 s p50 this is under 0.1%. No wait, round trip or file transfer is added. Recovery work (fence, journal read) runs only after a failed start. - Live restart-chaos wave and runner image: see the comments on this pull request. ## Risks - A false "not dispatched" proof would duplicate a prompt. It needs a lone, complete `received` record for this exact turn ID, read after the fence proves the sidecar is gone. `dispatched` is fsynced before the prompt write, and a failed write aborts the send. A failed `received` write removes the file, so it cannot leave a lone record. - Upgrade safety: saved provider state gains no field (its decoder denies unknown fields), and only lifecycles an older Runner already handles are written. An older Runner ignores the journal file. State saved by an older Runner has no journal, which reads as unknown and keeps today's behavior. A new Runner with an older sidecar never sees a `received` record, so no proof forms. - The changed sidecar schema changes the pack contract digest, so Daytona needs a new runner image. Older image packs are not reused. - A failed start now waits for the provider lifetime fence (at most 5 s, usually immediate) before it settles. This is only on a failure path. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
e33ede9a8a |
fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4398a723c0 |
fix(native): settle tool callbacks and holds that a server restart orphaned (#20628)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs send provider tool calls to the server as durable
semantic inputs. The server runs the tool and saves the result in the
controller journal
> - Close must prove that every admitted tool call has a saved result.
Otherwise it reports `native_session_settlement_incomplete` and the
runner holds a cleanup gate (`native_session_cleanup_quarantined`)
> - A server restart in the middle of a tool call leaves its input
without a result. The new controller asks the tool authority for a
receipt. Tools without a receipt (assigned app tools,
`connections_search`) return nothing, so the input stays pending forever
> - Runs that already failed this way kept no-replay holds for a turn
that had completed. The task stayed blocked with no Retry and no
continuation
> - Restarts are normal. Work that a restart interrupts must settle by
itself, not wait for a person
> - This pull request settles such a callback as interrupted, with an
unknown outcome, when its owner is provably gone, and settles the holds
that earlier releases left once their controller is gone. It never runs
the tool or the turn again
> - The benefit is that close drains after a restart, and tasks that
earlier releases blocked this way continue on their own
## Linked Issues or Issue Description
Refs #20572 (that PR fixed the presentation of this quarantine. Its
Risks section names this runner drain rule as a separate follow-up.)
Refs #20641 (it settles a resolved no-replay hold when a sibling action
saved the run's files. This PR covers a different case: the hold's own
recovery saved the files, and its run's turn completed. Both use the
same recovery receipt, `completedWorkspaceRecoveryCondition`.)
**What happened?**
A remote native run started an assigned app tool call. The server
restarted 3 seconds later. The new server adopted the runner, and the
provider turn completed about 9 minutes later. The tool call never got a
saved result. Close reported `native_session_settlement_incomplete` with
`runnerSuspended: true`, `providerDrained: false` and one pending
callback. The session quarantined, and the same-run recovery was refused
at admission. A second run showed the same shape with a
`connections_search` callback after a later restart.
The first run then had two resolved-but-blocked holds with
`preserve_without_replay_v1` and an unknown action outcome: the
cleanup-quarantine hold, and a continuation hold whose workspace
recovery saved the files. The task view said "Saved work recovered. Send
a message to continue." with no Retry and no continuation. The task
waits on a child task and stays blocked.
**Expected behavior**
After a restart, a tool call without a saved result settles
automatically. A durable receipt is replayed. Without a receipt, the
agent is told that the outcome is unknown. Close proves the drain, and
no cleanup gate is left. Holds that earlier releases left for a
completed turn settle without a person, and the task continues when its
own wait ends.
**Steps to reproduce**
1. Start a native run and call an assigned app tool.
2. Restart the server while the tool call runs.
3. Let the adopted turn complete, then close the session. Close reports
`native_session_settlement_incomplete` and quarantines. The task stays
blocked by the no-replay holds.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
ce1ee9e89f |
fix: stop prompts from encouraging subtasks (#20362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read runtime prompts, tool help, skills, and role instructions. > - Several of these texts tell agents to create child tasks as a normal next step. > - One accepted-plan prompt even forbids work on the current task. > - Task parentage is useful, but it does not need a prompt that encourages task creation. > - This pull request removes those prompts and keeps the task APIs available. ## Linked Issues or Issue Description Refs #10642. This change adjusts prompt wording; it does not add delegation-cycle guards. **Current behavior** Prompts tell agents to create children after plan approval, use children for long work, and create a task just to pass context. Onboarding requires a child even for a single request. **Proposed behavior** Continue work on the current task. Keep explicit assignments, optional task creation, parentage, dependencies, and approval gates. Use the separate-task recipes when the user requests that structure or when continuing existing work. **Reason and benefit** Reduce needless task trees without removing subtask support. Keep the patch limited to wording, existing test expectations, and generated catalog data. ## What Changed - Align accepted-plan prompts and remove direct child-creation advice from runtime and error text. - Make task tool descriptions neutral. Keep their fields, defaults, and handlers. - Update the main skill, onboarding, CEO instructions, company templates, specialist workflows, and matching guides. - Keep historical release snapshots intact. Refresh the explicitly seeded schema fixture's catalog digest; the server still verifies historical release snapshot hashes. - Refresh the affected generated catalogs, capability inventory, semantic contracts, and fixture manifest. ## Verification - Final tested head: `c6e304d0c105cdbfbe0b2ab0a233307fcd91a04b`. - All 66 check runs are complete: 60 pass, one is neutral, and five are skipped by CI conditions. Snyk also passes. No checks fail or remain pending. - [PR CI](https://github.com/paperclipai/paperclip/actions/runs/42846226554) passes typecheck, build, server/general tests, all eight E2E shards, Runner support, Runner Rust, and the aggregate verify gate. [Windows](https://github.com/paperclipai/paperclip/actions/runs/42846226546) and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42846226108) also pass. - [Greptile](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751419835) and [Paperclip review](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751454010) give this exact head 5/5. All six inline threads are resolved. Neither current summary has findings. - Merged master `bf0220d34a156c81853755c3d1365ddc527037de`. Resolved three generated-file conflicts with the canonical generators. - Local verification passes: 473 focused prompt, continuation, UI, catalog, runtime and native-execution tests; Runner TypeScript build; generated metadata; UI token gates; and diff checks. - The prior head `c22018cb0f` failed three SDK evidence tests because two source hashes were stale. Reproduced all three failures locally. The existing generator refreshed only the hashes for the denial-copy file and its test. All 53 affected evidence/source-pin tests then passed, including private-value rejection. No scanner logic, assertions, or runtime dependency pins changed. Current CI confirms the support suite passes. - A prior commit failed the Rust process reaping test. Its cause remains unproven. No Rust code or assertions changed. Later Rust runs pass; this does not establish the cause of the earlier failure. - Local preflight built shared/plugin SDK packages, then stopped because `cargo` is absent. Current-head CI supplies Rust and full build coverage. `pnpm verify --explain` selects broad required CI; full local suites were not repeated. ## Risks - Wording changes guide the model; they do not enforce a ban on subtasks. - Existing installed skill versions and saved custom instructions need their normal refresh process. This PR does not migrate company configuration. - Existing approved task assignments and approval gates remain in effect. ## Model Used - OpenAI GPT-6, through Codex, with reasoning, tool use, and code execution. The exact routed model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
faae6b919f |
fix(runner): explain task scope on missing tool inputs (#19774)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents read saved task documents through native tools.
> - Active-task tools use the task bound to the run. Other task tools
require a task UUID.
> - Agents sometimes call the second set with only a document key.
> - The validator rejects the call, but gives no instruction about the
active-task tool.
> - This pull request adds scope instructions to the tool schemas and
returns them for missing required fields.
> - Agents can correct a rejected call. Task selection and permission
checks stay in place.
## Linked Issues or Issue Description
**What happened?**
Two observed document reads sent only `{"key":"sentry-case"}` to
`read_task_document`. Each returned `Invalid tool input: / must have
required property 'taskId'`. Each later retry with a task UUID returned
without an error.
Pattern: [Task reads fail when agents omit
taskId](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/703d8c29-6a40-4aa2-b2c6-e2c5c36b5657).
Related PR: #18824 changed document key validation. It does not address
task scope.
**Expected behavior**
The tool contract must state which task it reads. A missing-field error
must include the schema owner's recovery instructions. A retry must
still pass validation and task permission checks.
**Steps to reproduce**
1. Call `read_task_document` through the MCP bridge with only a document
key.
2. Inspect the error sent to the caller.
3. Retry with an authorized task UUID. Confirm the returned document
belongs to that task.
### Hypothesis and thesis for a fix
Hypothesis: generic tool descriptions contribute to confusion between
active-task and explicit-task tools. An agent can also ignore a correct
schema. These tests do not measure the rate of model mistakes.
Thesis: Paperclip owns task scope instructions. Its five explicit-task
tools now describe the required UUID and the active-task alternative.
The two MCP transports return direct missing-property descriptions from
the admitted schema. The transports do not infer task IDs or apply
Paperclip task policy.
Proof: the same HTTP MCP regression fails on baseline
`f673bd3cb5b5b65ec51781e37355e168c3fe8ae4` and passes with the fix. It
still rejects before handler dispatch. An explicit retry reaches the
requested task. An isolated real API and PostgreSQL test checks the
saved revision, active-task selection, malformed UUIDs, and
cross-company denial through both transports.
## What Changed
- Add UUID and active-task guidance to `get_task`, `comment_on_task`,
and the three explicit-task document tools.
- Return bounded, distinct descriptions for up to three direct missing
required fields. Keep ordinary validation errors unchanged.
- Share this formatter between the Runner and OpenCode MCP bridges.
- Add transport regressions, real route and database checks, and tool
documentation.
## Verification
- Baseline red: both transport regressions return only the
required-property error and fail the recovery instruction assertion.
- Candidate green: 58 Runner and OpenCode transport and error formatter
tests pass on Linux with Node 24.22.2.
- Final-head server verification passed all 115 tests: 113 document-key
checks and two real API integration cases.
- The two local integration cases use real MCP HTTP, agent authority,
API routes, and disposable PostgreSQL. They read a selected other task
and its saved revision, read the active task separately, reject missing
IDs for all five tools, reject malformed UUIDs, and return 404 for a
foreign company task.
- Runner TypeScript typecheck and build pass.
- The full server typecheck passed on head
`03a37e9e50896416cfd45be33bf6dbe3e13169df` with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1 node scripts/typecheck-layers.mjs`.
The first run ended without a compiler diagnostic. Final-head CI passes
and Greptile is 5/5.
- Full preflight stops because cargo is absent. Rust checks were not run
locally.
- One existing delayed-body fetch test failed with a Node 25 fetch
runtime error. All transport tests pass on Node 24. The first
integration attempt used the wrong fixture return shape; the test was
corrected before the passing run.
- CI follow-up: the Runner test program reported four TS18046 errors
because `Response.json()` returns `unknown` under CI typings. Both tests
now declare the response type. `pnpm exec tsc -p tsconfig.test.json
--noEmit` passes, and both transport suites pass again (56 tests). Final
head `ffccebca19fe9e21a097829adc020679ffa7d86c` passes CI and Greptile
5/5.
- Review follow-up: the foreign task now has a saved document with a
distinct body. Both integration cases passed again after this test-only
change. The read must return 404 and must not return that body.
Current-head CI passes and Greptile is 5/5.
- Final CI:
https://github.com/paperclipai/paperclip/actions/runs/42352822739 . All
required jobs passed for `ffccebca19fe9e21a097829adc020679ffa7d86c`.
Greptile is 5/5 with no unresolved review thread.
- Real local API behavior is verified. External model behavior and
deployed recovery are not verified. No merge or deployment is included.
## Risks
- Missing-field errors now contain schema descriptions. The text is
limited to direct properties in the admitted schema, at most three
distinct descriptions and 1,000 characters each.
- Nested schemas and references do not get recovery text. Their
validation errors remain intact.
- Clearer instructions can help callers recover. They do not guarantee
that a model will choose the correct tool.
## Model Used
- OpenAI GPT-6 via Codex, with tool use and code execution. The exact
deployment model ID and context window are not exposed to this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
DottaandClaude
|
bf45700ca3 |
fix(native): make Retry work after a turn start cut off once Codex recorded the turn (#20516)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run sends `turn/start` through its runner. A control-plane restart can kill the runner and Codex while that request is in flight > - #20507 lets the runner resume by itself when Codex's thread history proves the turn was never created. When Codex did record the turn, the turn may have done work, so the runner still fails closed and nothing replays it > - That branch must still end in a Retry that works. The next run on the same conversation is refused at the runner-state fence before it asks for a runner. #20507 let Retry continue after that run by matching the refusal's error text > - Driving that branch end to end with the real runner exposed a second dead end. When the owner's runner dies, its provider-session close cannot prove suspension, so this server process can keep an operator-required cleanup quarantine for the task. Retry was then refused outright ("Send a new message") > - This pull request replaces the text match with a server-authored run-log event, lets Retry take the same verified-stop path as a new message, and adds a real-runner test of the whole branch: the cut-off owner fails closed, the follow-up is refused with structured proof, Retry is admitted, and the fresh Retry run completes > - The benefit is that this restart window cannot strand a task, and the Retry gate no longer depends on error wording ## Linked Issues or Issue Description **What happened?** Follow-up to #20507 (restart-chaos wave `w-20261110T033426-s108`, t03). #20507 left one item open: prove that a person's Retry produces a working run when the restart cut off `turn.start` after Codex recorded the turn. It also matched the fence refusal by its error text. **Expected behavior** After that cut-off, the owner stays failed closed and the turn is never started again by itself. A person's Retry starts a new run that completes. It never fails with `quarantine_owner_unverified` and never returns 202 with no run. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts`. 2. Without the new event, the test fails: the refused follow-up has no proof that it never launched, so Retry after it would wait on `process_identity_missing`. ## What Changed - `native-session-executor.ts`: when the runner-state fence refuses a run before launch (`prior_owner_*` or `legacy_prior_owner_*`), the executor records the server-authored event `native.prior_owner_fence_refused` with the closed refusal `reason`, then fails the run as before. A failed write leaves the run held (the safe side). - `native-local-process-stop.ts`: `nativeRunNeverLaunched` accepts a failed run that has this event (and still no launch, identity, `adapter.invoke` or runner-sourced event). It no longer reads the error text. - `explicit-native-continuation.ts`: when the task's in-process cleanup quarantine is operator-required, a Retry no longer returns `cleanup_quarantined` at once. Like a new message, it must first pass `verifyStoppedNativeSessionForContinuation`, which releases the quarantine and records `native.stopped_conversation_verified`. A native Retry always starts a fresh conversation (`forceFreshSession`), so it never resumes the interrupted turn. If the proof fails, Retry still returns `cleanup_quarantined`. - `verifyStoppedNativeSessionForContinuation` (`native-session-executor.ts`), three narrow additions: - a run with `native.prior_owner_fence_refused` may stand in for the saved state's owner, like a run refused by the cleanup quarantine (both were refused before launch); - when the session root is gone, it reads the fence's single `identity_indeterminate` quarantined copy of the owner's state (several copies stay ambiguous); - a provider startup attempt in phase `initialization_failed` whose direct child exited and whose process and process group are gone no longer blocks the proof. A same-run restore that fails closed leaves exactly this record. Those PIDs are also re-checked before the quarantine is released. - New real-runnerd integration test `native-cut-turn-start-retry.integration.test.ts` (real runnerd and fake Codex with `--hang-turn-start after`): 1. The owner run's `turn/start` hangs after Codex recorded the turn; the runner and Codex are killed. The owner's provider state keeps the ambiguous start, and the thread history keeps the turn. 2. The same-run resume restores the runner and fails closed with "cannot safely recover an ambiguous Codex turn start". Codex saw exactly one `turn/start`. 3. A follow-up on the same conversation is refused with `prior_owner_*`, never `quarantine_owner_unverified`. It records `native.prior_owner_fence_refused`, launches nothing, and `nativeRunNeverLaunched` is true for it. 4. The owner's operator-required in-process cleanup quarantine is pinned (whether the real close lands there is timing-dependent; about half of the local runs did). 5. Retry of the refused follow-up passes `admitExplicitNativeContinuation` (not a dry run) with no block reason, releases the quarantine and records `native.stopped_conversation_verified`. 6. The Retry run, prepared with a fresh conversation as task Retry does (`forceFreshSession`), uses a different runner-state root and completes with exit code 0. Codex saw one new `thread/start` and one new `turn/start`; the cut-off turn was never started again. - `explicit-native-continuation.test.ts`: the fence rows now insert the event. A new row shows that the refusal text alone keeps the run held. The "worker loss (retry)" case now expects a verified Retry to be admitted like a new message. - `native-recovered-owner-followup.test.ts`: a fixture event moves to a free sequence number, because the refused follow-up now writes its own event. - Docs: `doc/run-log-events.md` (new event) and `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Red/green: - Without the event write, the integration test fails (no fence event, so the follow-up is not proven unlaunched). - Without the gate and proof changes, it fails with `expected [ 'cleanup_quarantined' ] to deeply equal []`: Retry is refused while the owner's quarantine is held. - With all changes, it passes. - `vitest run` of `native-recovered-owner-followup`, `explicit-native-continuation` (427 tests), `quarantined-task-retry`, `issue-confirm-run-stopped-routes`, `native-finalization-recovery`, `task-interruption-query`, `execution-projection`, `chat-run-publications`, `closed-native-receipt-accounting`, `agent-live-run-routes` and the new integration test pass. - Server `tsc --noEmit` passes. - Runner code and the pack `contractDigest` do not change. ## Risks - Low-medium risk. The new event is written only on the fence refusal path, which already fails the run. It is local run-log data (no Telemetry or OpenTelemetry). - Retry after an operator-required cleanup quarantine is a behaviour change: it was always refused, and now it is admitted only after the same exact stop proof a new message needs. That proof checks the owner's runner stop record, every recorded provider and startup PID and process group, and the unchanged state fingerprint before it releases the quarantine. - A run refused by a build between #20507 and this change has only the error text. It stays held on `process_identity_missing` until a person sends a new message or the board confirms the stop. #20507 has not shipped in a release yet. - The test models the restart in-process: it kills the runner and Codex and calls the executor again. It does not drive the HTTP Retry route; the route's own behaviour (fresh conversation, admission gate) is covered by existing tests and by `nativeRunNeverLaunched` here. - Open decision, unchanged: a turn Codex recorded before the restart is never replayed or adopted. Continuing is a person's decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20507 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
a09093a5c5 |
fix(native): resume a run whose turn.start a restart cut off before Codex created the turn (#20507)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run sends `turn.start` to its runner, and the runner sends `turn/start` to Codex. The runner marks the start "ambiguous" before it sends the request > - A control-plane restart can kill the runner and Codex while that request is in flight. After the restart, neither journal says whether Codex created the turn, and the resumed thread has no active turn > - The restored runner then failed closed on every same-run resume with "cannot safely recover an ambiguous Codex turn start without an active replacement turn". The answer never reached the agent > - The next run on the task was refused at the runner-state fence (`prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`) before it asked for a runner. Retry after that run then waited forever on `process_identity_missing` for a process that never existed > - This pull request lets the runner prove from Codex's own thread history that the start never created a turn, and then resume and start the turn once. It also lets Retry continue after a run that the runner-state fence refused before launch > - The benefit is that a turn start cut off by a restart continues by itself when Codex never saw it, and a person's Retry is no longer a dead end ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T033426-s108`. Command: `--env local --adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Task t03 (Q2): the answer run `92a57f50` resumed after the first restart. A second K1 restart landed 0.25 s after `session.resumed`, while its `turn.start` was in flight. Each same-run resume then failed with `PRP command session.snapshot failed: cannot safely recover an ambiguous Codex turn start without an active replacement turn`. When the prerequisite finished, the next run `0e6aa9d1` failed before launch with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The accepted Retry (202) never created a run: the task stayed on `retryWait.reason=process_identity_missing` until the wave timed out (`R3:retry_no_new_run`). **Expected behavior** If Codex never created the turn, the run resumes and starts it once by itself. If that cannot be proven, a person's Retry starts a new run instead of waiting forever. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-turn-start.test.ts` on master (case "before" fails with the wave error). ## What Changed - Runner (`codex_provider.rs`, `provider_backend.rs`): when the restored provider state has an ambiguous turn start, no active turn before the loss, and the resumed thread has no active turn, the runner reads the newest turn of the thread (`thread/turns/list`, limit 1, metadata only, the same request `read_thread` already sends). If there is no turn, or the newest turn is one Paperclip already settled, Codex never created the turn. The runner clears the ambiguity, emits a `harness.diagnostic` with code `ambiguous_turn_start_not_created`, and restores an idle session. The old `turn.start` stays `execution_indeterminate` and is never executed again. The same-run resume then starts the run's turn once, through the existing "recovered session without an active turn" path. - Any other history (a turn Paperclip never settled, an in-progress turn, an incomplete page, a non-Codex provider) keeps today's fail-closed error. The probe runs only on this recovery path. The normal turn path does no new work before `turn.start`. - Server (`native-local-process-stop.ts`): `nativeRunNeverLaunched` also accepts a failed native run whose error is the runner-state fence refusal `runner_state_identity_mismatch: prior_owner_*` (or `legacy_prior_owner_*`). The executor throws it only while it migrates another run's saved state, before it requests a launch. The existing evidence check still applies: any launch request, identity record, `adapter.invoke` or runner event keeps the run held. Other runner-state errors (for example `authority_indeterminate`) stay held. - Fake Codex (`fake-codex-app-server.rs`): keeps the thread's turn history in a `<state-file>.turns` sidecar, so `thread/turns/list` returns history like real Codex (newest first, honours `limit`). New `--hang-turn-start <file>` holds `turn/start` "before" or "after" the thread records the turn, to model a kill in flight. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-turn-start.test.ts`. A real runner and fake Codex start a turn; both processes are killed while `turn/start` is in flight. "before": the resumed run completes, the journal holds the old `turn.start` (`execution_indeterminate`) and one new completed `turn.start` with the same task text, and Codex saw exactly one more `turn/start`. "after": recovery still fails closed and no new `turn/start` is sent. - Rust: `ambiguous_turn_start_that_never_created_a_turn_recovers_and_starts_once` and `ambiguous_turn_start_that_created_an_unsettled_turn_still_fails_closed`. Both rewrite saved provider state the way a master runner wrote it, so they also cover state saved before this change. - `explicit-native-continuation.test.ts`: five new rows for the runner-state fence. Admitted: refused before launch (with and without a coordinator). Held on `process_identity_missing`: refused after a launch request, refused after a runner event, and a run that failed its own `authority_indeterminate` check. - `docs/durable-recovery.md`: describes both behaviours. ## Verification - Red/green, real runnerd: on master (the staged master runnerd), `runnerd-codex-transport-cut-turn-start.test.ts` "before" fails with `PRP command runner.drain failed: ... cannot safely recover an ambiguous Codex turn start without an active replacement turn`, the wave error. With this change, both cases pass. - Red/green, Rust: without the `provider_backend.rs` change, `cargo test -p paperclip-runner-core --test codex_provider ambiguous_turn_start_that` fails the "never created" test. With it, both pass. The full `cargo test -p paperclip-runner-core` passes. - Red/green, server: without the `native-local-process-stop.ts` change, the two "refused at the prior-owner runner-state fence" rows fail with `process_identity_missing`. With it, all 14 rows pass. - Related suites pass: 11 Runner transport files, 419 tests (`runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `-cut-turn-start`, `-settlement`, `-suspension`, `runnerd-restart-continuation`, `runnerd-saved-continuation`); server `explicit-native-continuation`, `quarantined-task-retry` and `issue-confirm-run-stopped-routes`, 3 files, 470 tests; full `cargo test -p paperclip-runner-core`. - `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit` and server `tsc --noEmit` pass. - Pack `contractDigest` is unchanged: `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). - Live proof: the wave command above on this branch. Results follow in a comment. **No duplicate provider work.** The runner starts a new turn only when Codex's own history proves the cut-off start created nothing. The old command is never executed again (the runner refuses an `indeterminate` command). A recorded turn that Paperclip never settled keeps the run failed closed. ## Risks - Low-medium risk. The runner change runs only while it restores a session whose saved state has an ambiguous turn start and no active turn. Before this change, that run could never continue. - The proof relies on Codex recording a turn in the thread before it runs anything. If Codex accepted the request but died before it wrote the turn, the runner treats the start as not created and starts the turn once. Codex writes the user turn to the thread when it starts the turn, before it asks the model for output, so no model output or tool call is expected without a recorded turn. - If the thread's newest turn is not in the runner's settled set (for example after a provider identity epoch rollover forgot older identities), the runner keeps failing closed, as today. - **Open decision (not changed here):** when Codex did record the cut-off turn, it may have done work. This pull request keeps the run failed closed and does not replay or adopt that turn. Continuing is a person's decision through Retry or a new message. - The server change trusts the executor's own error text `runner_state_identity_mismatch: prior_owner_*`, together with the absence of every launch and runner event. A run that requested a launch keeps every stop proof. - **Runner release:** the pack `contractDigest` does not change. Local runners use the runnerd built with the app, so the fix applies with this release. - Not covered: the related `authority_indeterminate` variant in the same wave (t04, Q1) is a separate cause; its Retry already worked in that wave. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20477, #20482 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
914e3ae14f |
fix(runner): settle ACPX turns when the cancel reply is lost (#20498)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the native Runner and the ACPX sidecar > - A saved question or a board Stop cancels the active turn with `turn.cancel` > - The Runner waited 30 s for the sidecar's reply to that cancel, and the Runner serves controller commands one at a time > - When that reply was lost, the controller's shutdown commands timed out first, so the run failed and the task needed a human Retry > - This pull request bounds that wait, keeps the transport open, and lets the provider's own turn terminal settle the turn > - The benefit is that a lost cancel reply on Claude settles the same way it already does on Codex (#20336) ## Linked Issues or Issue Description Refs #20336 (Codex equivalent: settle a saved handoff after verified shutdown). Related, no overlap: #20477, #20482, #20483. **What happened?** The restart-chaos harness dropped the ACPX sidecar's reply to the Runner's `turn.cancel` (fault PV2, target `interrupt_ack`). A Claude task that saved a question (Q1) failed with "The waiting state is saved; confirming shutdown of the previous execution is still pending". Its next attempt failed `native_session_cleanup_quarantined`. A Claude task stopped by the board (S1) ended `blocked`. Retry was accepted but started no run. Cause, in `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs` (`interrupt_turn`, master line ~800): the cancel used the ordinary sidecar request with the 30 s bound from `acpx_provider_backend.rs:539`. While it waited: - `turn.stop` and `runner.suspend` stayed queued. The controller's 15 s governed drain plus its 10 s close grace ran out first (`runner did not durably suspend before checkpoint`, suspend command `pending`). - At 30 s, `fail_closed` killed the sidecar and dropped the provider's buffered turn terminal. The poll path then recorded `turn.failed` (`acpx_provider_transport_failed`) and `provider_exit_unconfirmed` (`acpx_provider_backend.rs:2716-2738`). `turn.stop` returned `already_settled`, and `runner.suspend` refused with `ACPX provider lifetime cleanup is not yet proven` (`acpx_provider_backend.rs:2613-2621`). Codex passes the same fault because its Runner does not block on the interrupt reply. **Expected behavior** A lost cancel reply must not fail the run. The saved question must settle as `succeeded`. A board Stop must end the run as cancelled with a proven suspension, so the next turn can start. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4` Failing wave on master: `w-20261110T060120-s207` (t04, t05). **Deployment mode** Native Runner with the Claude ACPX profile, local. Hosted sandboxes run the same Runner code. ## What Changed - `AcpxSidecarTransport::request_turn_cancel`: sends `turn.cancel` and waits only to a caller deadline. On a missing reply it keeps the transport usable and retains the request ID, so a late reply is skipped in both the response and the event-poll paths. A rejected reply and transport errors keep their existing fail-closed result. - `AcpxProviderSession::interrupt_turn`: waits at most 5 s (or the shorter configured request bound) for the cancel reply. On a missing reply it retires the runtime from warm reuse and returns. The provider's turn terminal still arrives through polling, and `turn.stop` still reaps a provider that never stops. - `AcpxProviderSession::deliver_tool_result`: a board Stop can land while a semantic tool runs. The session's own `turn.cancel` then retires the call in the sidecar before its result arrives, and the turn terminal may still be buffered. The sidecar rejects the result as stale (`ACPX_TOOL_CALL_STALE`). When this session sent `turn.cancel` for that turn, the Runner now keeps the durable result, like a result for an already closed turn. Every other rejection still fails closed. (`AcpxSidecarTransport::request_tool_resolve` reports whether a rejection was the stale-call case.) - A repeated interruption while a cancel is unanswered sends nothing and returns at once (review finding). - `fake-acpx-sidecar`: new `--cancel-reply lost|late` flag. After that cancel, `tool.resolve` is rejected as stale. The provider ends the turn, and the reply is dropped or written just before the next request. - Three regression tests (below) and a short section in `packages/paperclip-runner/docs/durable-recovery.md`. No TypeScript, server, schema or protocol change. The normal start path is unchanged; nothing waits before turn start. ## Verification Red/green (red runs used the master versions of `acpx_provider_session.rs` and `acpx_sidecar_transport.rs`): - `cargo test -p paperclip-runner-core --test acpx_warm_attachment a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner` - Red: after 30.4 s, `ACPX turn interrupt failed: ACPX sidecar request timed out at turn.cancel` (the live error). - Green: `turn.interrupt` returns in about 5 s, then `turn.interrupted` and a `cancelled` `run.terminal` (no `turn.failed`), `turn.stop`, and `runner.suspend` completes. - `cargo test -p paperclip-runner-core --test acpx_provider_turns lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport` - Red: `interrupt_turn` fails at the request bound. - Green, for a lost reply and for a late reply: the terminal is polled, warm reuse is refused, and the next request (`session.close`) succeeds after the late reply arrives. A late reply causes no second settle and no second prompt. - `cargo test -p paperclip-runner-core --test acpx_provider_turns a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport` - Red (guard disabled): `ACPX sidecar command tool.resolve was rejected (retryable=false, classification=provider_tool_call_retired)`, the same error as live wave task t03 below. - Green: the result is kept, the pending tool clears, and the turn terminal is polled after it. - `cargo test -p paperclip-runner-core` (whole crate): all pass. Existing `turns-tool-resolution-rejected` coverage still fails closed when no cancel was sent. `cargo fmt --check` passes. - Pack contract digest: unchanged (`sha256:3164357d…` before and after, from `scripts/contracts/contract-fingerprints.mjs`). - Live wave 1, `w-20261110T072706-s211` (commit `d91bff1e9f`, the first fix only), `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --tasks 4`: 3 of 4 pass. Both Q1 tasks pass (the asking run `succeeded`, no failed run, answer in the brief). One S1 passes. S1 task t03 failed: Stop landed during a `write_task_document` call and hit the stale tool-result error above. The second commit fixes that case. - Live wave 2, `w-20261110T082702-s212` on head `76dd57515d`, same flags: every task passes R1–R6 and U1. Both Q1 tasks pass with the fault applied (asking run `succeeded`, no failed run). S1 t01 passes with the fault applied (Stop took effect, task `done`, no Retry). S1 t03 fails only `provider_fault_applied` (`applied=0`): the agent finished before the Stop window, so nothing was injected and the task proves nothing either way. - Live wave 3, `w-20261110T091057-s213` on head `76dd57515d`, same flags: the same result. t01 (S1), t02 and t04 (Q1) pass with the fault applied; t01's Stop took effect (`status=cancelled`), task `done`, no Retry. t03 again fails only `provider_fault_applied` (`applied=0`, the run finished before the Stop window); all its R1–R6 and U checks pass. - Across waves 2 and 3 on the final head: 6 of 6 tasks with the fault applied pass (4 Q1, 2 S1). No product rule failed. The two misses are harness timing (no injection), not product failures. ## Risks - A healthy cancel reply that takes longer than 5 s now retires the warm runtime. The next turn starts a fresh provider process for that session. The observed healthy reply time is about 1 s. - If the provider never stops and the reply is lost, the turn stays active until `turn.stop` terminates the provider and proves the lifetime fence. If that proof fails, the existing settlement-pending and quarantine states are unchanged; this PR adds no success path without proof. - Runner binary change: hosted Daytona images need a normal image rebuild to pick up the fix. The pack `contractDigest` does not change, so existing image packs stay reusable. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
3e572475c8 |
fix(native): re-attach an answer run whose run.attach a restart cut off (#20477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person answers an agent's saved question, Paperclip starts an answer run. For a per-turn session, that run starts a new runner and sends `run.attach` to bind the saved conversation to the run > - The runner journals each command before its effect. A restart that kills it inside that window leaves `run.attach` with an unknown outcome. The restored runner settles it as `execution_indeterminate` and never executes it again > - The controller's same-run resume waited on that old command, so every resume failed with "PRP command run.attach failed". The answer never reached the agent. A Retry or a stall followed > - A kill during the attachment's provider launch can also leave a startup fence that no later runner retires, so every later command fails with "provider startup ownership remains unadmitted" > - This pull request attaches again with a new command when nothing after the cut-off attachment started provider work. It also lets the runner retire that orphaned startup fence > - The benefit is that an answer given just before a restart continues on its own ## Linked Issues or Issue Description **What happened?** Local restart-chaos journey wave `w-20261110T054406-s108` on master `2c2186706e`. Command: `--adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Result: 1 of 4 passed. - t02 (Q1, K1 2.7 s after the answer). Answer run `0b357a1f` resumed after the restart. Each resume failed with `native_session_recovery_failed: PRP command run.attach failed: execution_indeterminate`. Then "Automatic recovery stopped", and the task needed a Retry. - t01 (Q2, K1 1.1 s after the answer). Answer run `82c60af9` failed the same way on three same-run resumes. Its provider state kept `lifecycle: "prepared"` with a `spawned` startup attempt for `run.attach`, and that provider process was gone. Each later drain and suspend failed with "provider startup ownership remains unadmitted". The successor run then refused with `retirement_refused=journal_commands_unsettled(run.attach.failed)`. The accepted Retry created no run (`retry_no_new_run`). The earlier wave `w-20261110T033426-s108` hit the same window at different points. It showed `authority_indeterminate` (Q1) and an ambiguous `turn.start` stall (Q2). See Risks for what this pull request does not cover. **Expected behavior** An answer run that a restart interrupts before it starts provider work resumes by itself. It does not need a Retry. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-attach.test.ts` on master. Both cases fail with the error from the wave. ## What Changed - `runnerd-codex-transport.ts`: the same-run resume waits on the recovered `run.attach` as before. If that wait fails, the new `cutOffRunAttachmentPayload` checks the journal. It queues one new `run.attach` with the same payload, and waits on it, only when all of these are true: - the latest `run.attach` has an unknown outcome (`indeterminate`, or `failed` with `execution_indeterminate`); - it is a cold attachment (no `paperclipNextAuthority`), with no warm transition and no staged attachment; - every later command is recovery maintenance, as the control plane's shared `isRunnerRecoveryMaintenanceCommand` defines it (`session.snapshot`, `session.goal.get`, `runner.drain`, `runner.suspend`, `runner.shutdown`, `turn.stop`); - no committed event shows provider work (`turn.accepted`, `turn.started`, `item.started`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created` or `mcp_app.tool_input`). Any other failure is thrown unchanged. - `provider_backend.rs` (Runner): `retire_orphaned_startup_attempt` also treats a `prepared` state with a saved thread and a nonzero process generation as restorable. `attach_run` already documents this state as the settled checkpoint that `run.attach` restores. The other fence rules do not change. The fence is retired only before the spawn, or when the recorded process and its group have exited. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-attach.test.ts`. A real runner and the fake Codex server run an asking run, then a cold answer-run attachment. The test kills both processes and rewinds both journals to "`run.attach` journaled, outcome not recorded", then resumes. One case has a startup fence and one does not. - A unit test for `cutOffRunAttachmentPayload`. It covers each maintenance command, and the cases where it must not re-attach (a definite answer, a pending command, a warm or staged attachment, any work command, a turn event). - The Rust orphaned-startup test also covers `prepared`. A new test shows that a `prepared` state with no saved thread keeps its fence. - `docs/durable-recovery.md`: describes both behaviors. ## Verification - Red/green, real runnerd: - On master, `runnerd-codex-transport-cut-attach.test.ts` fails both cases with `PRP command run.attach failed: ... execution_indeterminate`, the error from the wave. - With only the transport change, the case without a fence passes. The fence case fails with `provider startup ownership remains unadmitted`, the t01 error. - With both changes, both cases pass. - Red/green, Rust: `cargo test -p paperclip-runner-core --lib orphaned` fails `restart_retires_a_spawned_startup_whose_provider_process_exited` for `prepared` without the runner change. With it, all 10 tests pass. - Related transport suites, 8 files and 387 tests, pass: `runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `runnerd-restart-continuation` and `runnerd-saved-continuation`. `tsc --noEmit` for the Runner package passes. - Live proof: the same wave command on this branch. Results follow in a comment. **No duplicate provider work.** The new attachment is sent only when the journal proves that no provider work followed the cut-off one. No command after it can start or steer work: only the shared recovery-maintenance commands are allowed, and none of them starts a turn. No turn, item, tool-input or runtime-request event is committed. `run.attach` itself starts no turn. In the Codex and ACPX runners, `attach_run` refuses to attach unless the session is settled. That means no active or ambiguous turn, no pending tool call and no live active goal. The old command is never executed again, because the runner refuses an `indeterminate` command. The new attachment is a new command with a new ID. ## Risks - Low-medium risk. The new path runs only after a same-run resume has already failed on an unknown-outcome attachment. Before this change, that run could not continue. - If an attachment's partial effect had changed provider state, the runner's own settled-session check refuses the new attachment. The run then fails as it does today. - **Runner release:** the Runner change does not change the pack `contractDigest`. It is `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). Local runners use the runnerd built with the app, so the fix takes effect with this release. A remote image pack (for example Daytona) that was built before this change keeps the old fence rule until a new image or pack is built and qualified. No release or pin bump is part of this pull request. - **Mixed versions during an upgrade:** - New server with an old runner: the server re-attaches. A run without a startup fence continues. With a fence, the old runner refuses the new attachment, the same failure as today. - Old server with a new runner: the runner retires the fence, but the old server still fails on the unknown-outcome attachment, as today. - Neither mix adds a new failure. Saved state from older builds needs no migration. - Not covered here, and still open: - A `turn.start` with an unknown outcome where the resumed thread has no active turn (wave `w-20261110T033426-s108` t03). The runner deliberately fails closed for that case. - A warm-runner attachment that may already have been delivered (`authority_indeterminate`). - A Retry that waits on `process_identity_missing` for a run that failed before any process started. A separate pull request will cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20146, #19784, #20148 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
79bf733012 |
fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner keeps a lifetime fence until the original ACPX provider stops. > - A recovered Runner must prove that fence is free before it clears saved cleanup state. > - Suspension and other recovery paths already wait for transient listener holders. Recovered shutdown used one immediate probe. > - That difference made parallel Rust tests fail after they dropped their original listeners. > - This pull request uses the existing bounded proof for recovered shutdown and removes the repeated test retry. > - The benefit is one cleanup rule, with the same fail-closed proof and no new work on normal task dispatch. ## Linked Issues or Issue Description Fixes: #20210 Thanks to @genius-0963 for the test-only fix proposed in #20321. That related alternative adds a test-only shutdown retry. This change applies the existing bounded cleanup proof in the shared shutdown path and removes the same retry pattern from native-provider integration tests. It also tests a single shutdown call while the original quorum is still held. ## What Changed - Use `await_provider_lifetime_fence` for a recovered executor with no live session and unconfirmed provider exit. Delete the unused one-shot helper. - Keep the acquired quorum held through the existing durable state write. A failed proof leaves `providerExitUnconfirmed` set. - Give the shared timeout message wording that applies to shutdown and suspension. - Extend the recovery test with a worker-start signal. Hold the original listeners until the single shutdown call is waiting, then release them and assert successful cleanup without a provider session. - Use the same bounded proof in the retired-quorum fixture test. Remove the duplicate retry loop from the native-provider test helper. - Document the bounded recovery cleanup wait. ## Verification - macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only targeted checks ran locally. - `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`: passed. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on the rebased head (5.21 seconds). The initial head also passed five more parallel-test runs (45 tests per run), with no failure. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test native_provider_backend acpx_`: 14 passed on the rebased head (9.43 seconds). These tests use the real Rust executor and controlled sidecars. - Negative control: restore only shutdown's original one-shot proof. The new regression fails at `shutdown must wait while the original lifetime still holds a quorum`. Restore the fix and the affected module passes. - Independent parent verification on current commit `368fc814f7`: the three exact recovery, held-quorum, and retired-quorum tests passed (3 tests, 5.06 seconds). Independent source review confirmed the unchanged three-file scope and found no actionable issue. - The active-turn recovery test still holds a live quorum through the five-second bound. It checks that shutdown refuses cleanup, saved state remains unconfirmed, and the provider never starts. It then releases the quorum and confirms cleanup. - Standard Greptile scored current commit `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads. - Initial GitHub general-server shard 7 failed in unchanged `remote-workspace-nested-verification.test.ts`, at its 15-second wait for a child-process import checkpoint (expected one row, got zero; both Vitest attempts). [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815). That fixture runs Node, PostgreSQL, Git, and storage migration; it does not run the native Runner or the changed Rust cleanup. The cause of the checkpoint timing miss remains unknown. The focused retry passed on the unchanged source ([successful retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)). This proves recovery, not the cause of the original timing miss. GitHub kept the reusable-workflow aggregate failed after that job-only retry and rejected both a final-gate retry and a failed-job retry. A subsequent run-level retry materialized only the gate and aggregate jobs; its aggregate inputs reported prior jobs as `abandoned`, so it failed without a new test failure. A new `full-ci` label event requests a fresh workflow graph on the same commit. The reason for those rerun-state results is not established. - The fresh `full-ci` run [42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208) then lost several runners mid-job. The maintained [runner-loss recovery 42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525) classified all four failed jobs as runner loss: typecheck, E2E shards 2 and 8, and server shard 4. The evidence was a shutdown log or lost-communication annotation. It retried the failed jobs together; canceled sibling jobs were also requeued. This is runner-interruption evidence, separate from the earlier checkpoint assertion. - The maintained recovery attempt again failed the same partial-checkpoint assertion in unchanged `remote-workspace-nested-verification.test.ts`, twice ([job 129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)). Neither checkpoint-failure job logged runner shutdown or lost communication; their resource summaries reported no significant constraint. The child phase at the timeout remains unknown. No further blind retry or TypeScript change is included here. Related startup-handshake work exists in #20339; its controlled-delay evidence establishes a deadline-boundary defect but does not establish the phase of these original CI failures. - Rebased onto current master after upstream #20339 merged. That separate change fixes a demonstrated child-startup/checkpoint-deadline boundary. It does not prove which child phase caused the earlier zero-checkpoint assertions. This PR still changes only its original Rust and documentation files. Fresh GitHub CI on the new base will establish recovery, not retrospective cause. - Full local tests and a local release build were not run. Current-head [PR CI 42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411), [Runner Windows 42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803), and [Docker Runner 42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601) all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check list has 59 successes and 5 intentional skips. GitHub supplied broad typecheck, build, and optimized Rust coverage. This is new-base recovery; the cause of the original checkpoint assertion failures remains unproved. ## Risks - A genuinely unconfirmed original provider can now hold recovered shutdown for up to the existing five-second proof deadline. The proof returns immediately when two candidate ports are free. - The changed branch requires both no in-memory provider session and unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch keep their existing paths. Outer controller close deadlines remain authoritative. - State saved by the previous release carries forward unchanged. No schema, launch identity, profile digest, protocol, or retry-budget change. - No live provider session or hosted deployment was tested. The tests cover cleanup proof, refusal, durable state, and controlled provider execution. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
09412e652a |
fix(runner): allow warm release to drain after provider restart (#20290)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm Runner sessions park after a turn so follow-up tasks can claim the same process. > - Claude restarts its ACPX provider after the controller receives the terminal result. > - That restart can delay the drain receipt beyond the release path's one-second limit. > - A failed drain prevents parking, so the idle timer later closes the retained Runner. > - This pull request uses the existing warm admission budget for release drain. > - The benefit is retained Runner reuse without moving provider restart onto task dispatch. ## Linked Issues or Issue Description Fixes: #20225. Refs #20190 and #20145. Related performance follow-up: #20224. ## What Changed - Align the release drain timeout with the existing 30-second warm admission budget for every provider. Close paths keep their explicit remaining deadlines. - Keep the existing immediate completion when the drain receipt proves settlement. Add no sleep, retry, provider branch or dispatch work. - Add a real Runner regression with a fake ACPX sidecar whose replacement start takes 1.5 seconds. One release call must park the Runner, then a follow-up must claim the same Runner process. - Document the release drain budget. ## Verification - Targeted command in `packages/paperclip-runner`: `vitest run src/live/runnerd-codex-transport-release-resume.test.ts src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed** across 3 files, 157.78 seconds. - Independent parent verification reran the slow-restart and ordinary cached cases: **2 passed**. Source review confirmed that release is the only caller using the changed default; close callers supply explicit deadlines. - Negative control: with the original one-second timeout, the new test fails because the single release call returns `false`. The fixed code passes the new case. - Built only the current-source Runner and two fixture binaries required by these tests. The first sandboxed invocation could not bind its loopback listener (`EPERM`); the test ran with loopback access afterward. - Full GitHub PR workflow [42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993) passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard Greptile scored this same head 5/5 with no review threads. Full repository checks ran in GitHub CI, per the requested local test scope. No live Claude session or hosted deployment was tested. ## Risks - An unresponsive Runner can now hold release drain for up to the existing 30-second warm admission limit. A healthy Runner returns as soon as its receipt arrives. The provider restart stays at turn end; task dispatch gets no fixed delay. - Saved state, protocol and provider identity formats do not change. Previous-release state carries forward under the existing validation. - The controlled fake-sidecar test proves the restart timing race and process reuse. It does not qualify live provider latency or a hosted rollout. ## Model Used - OpenAI GPT-6 through Codex, with repository tools, code editing and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
Devin FoleyandPaperclip
|
2d4f80504e |
fix(runner): retain exact ACPX checkpoints after rejected turn starts (#19762)
Preserve the exact idle ACPX conversation when a matching turn.start response rejects work with retryable SESSION_RESUME_REQUIRED. Require verified identity, no pending work, confirmed process cleanup and the provider lifetime fence before normal drain and suspension can publish a reusable checkpoint. Unknown acceptance and arbitrary closed states remain closed. The real Runner regression completes the next turn on attempts 2 and 3, requires successful terminal status for the exact submitted turn, and checks expectedIdentity on both resume requests. Controlled wire faults reject failed, cancelled and missing-identity recovery. Validation: 193 affected Rust tests on unchanged Rust source/fixtures; 134 recovery integration tests, 14 environment-gated skips; 21 Claude effort tests; all server typecheck layers. Exact-head CI: 59 successful checks and statuses, 6 intentional skips. Greptile Apex 5/5, no unresolved findings. Hosted deployment and task acceptance remain separate. Fixes: #19751 Refs: #19366 Refs: #19745 Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
DottaandClaude Opus 5.5
|
20fc37778f |
fix(runner): keep settled OpenCode sessions across a controller proxy rebuild (#19641)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps an OpenCode conversation in durable state,
so a task can continue after a control-plane restart.
> - Saved OpenCode state binds one digest of the launch profile: the
Node command, the proxy bundle and the OpenCode executable, each by path
and SHA-256.
> - The Node command and the proxy bundle are controller runtime. A
rebuild or deploy between a hard kill and recovery replaces their bytes.
> - The recovery probe then refused a settled conversation with
`provider_state_incompatible`, although the OpenCode release and the
conversation were unchanged.
> - ACPX already re-binds a settled session after a Runner upgrade
(#18879). OpenCode did not.
> - This pull request records the OpenCode release digest in saved state
and re-binds a settled session when that release is unchanged.
> - The benefit is that a task continues automatically after a restart
that also changed the controller build. A real release change still
fails closed.
## Linked Issues or Issue Description
Refs #19558, #19501, #18879.
**What happened?**
A restart-chaos wave killed the control plane with SIGKILL while an
OpenCode run was settled between turns. The first recovery attempt
restored the session. The controller's runner `dist` was then rebuilt in
the same checkout. The next recovery attempt failed in the
`runner.drain` resume probe with `OpenCode runner launch profile changed
across durable recovery` (`native_provider_state_incompatible`). The
saved state matched the original proxy bytes exactly. Only the proxy
bundle bytes had changed. The OpenCode executable, the paths and the
model were the same.
**Expected behavior**
A settled OpenCode session continues after a restart when only
controller runtime bytes changed. A changed OpenCode release, or a
session with work in flight, fails closed. The existing disposition then
offers one Retry that continues in a fresh provider session.
**Steps to reproduce**
1. Start an OpenCode run and let it settle between turns.
2. Kill the runner and the controller.
3. Replace the qualified proxy bundle with valid new bytes (rebuild or
deploy).
4. Recover the run. The resume probe fails with
`provider_state_incompatible`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
7c673e695f |
fix(chat): name tasks by identifier, not UUID, in tool rows and agent text (#19635)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - New users meet Paperclip through the onboarding chat with their
first agent, and through task briefs that agents write
> - A browser walk of onboarding found internal handles and
control-plane jargon in those user-facing surfaces
> - A tool row showed `Read taskId: e3ecf9bc-…`, an agent-written brief
cited `task … (id 2b330993-…)`, and a Codex agent told a new user about
"receipts", "execution policy", and a "rejected policy patch"
> - Raw UUIDs and mechanism narration make the product look broken and
give the user nothing to act on
> - This pull request makes tool rows name a task by its identifier, and
tells agents to reference identifiers and to describe outcomes, not
mechanics
> - The benefit is that a new user sees "Read <task identifier>" and
plain progress updates
## Linked Issues or Issue Description
**What happened?**
In the task chat, a collapsed tool row rendered the raw call argument
`taskId: <uuid>`. An agent-written task description included a raw task
UUID next to the task identifier. A Codex chief of staff narrated
internal mechanics (receipts, execution policies, a rejected policy
patch) to a new user in chat.
**Expected behavior**
Tool rows name the task by its identifier. Agent-written text links
tasks and documents by identifier. Agents tell the user what happened,
what it means, and the next step.
**Steps to reproduce**
1. Run onboarding with the Codex API runner and accept a single-task
proposal.
2. Watch the handoff in the chief-of-staff chat.
3. See the task lookup tool row and the narration. With the Claude API
runner and the plan path, open a child task created from the plan and
read its description.
## What Changed
- `ui/src/components/task-chat/transcript-adapter.ts`: a tool row's
target never contains a UUID argument. A `taskId`/`issueId` UUID is kept
as `taskRefId` on the tool item.
- New `ui/src/components/task-chat/task-reference.ts`:
`useTaskReferenceLabel` reads the task from the shared issue detail
query cache and returns its identifier. It returns "task" until the task
resolves or when it cannot.
- `TaskChatRunnerActivityGroup` and `TaskChatToolCard` show that
identifier as the row subject. They load the task only for rows that
have a task reference.
- `skills/paperclip/SKILL.md` (Comment Style): two new required rules.
"No raw IDs in prose" and "Describe outcomes, not mechanics".
- First-task onboarding skill: a "Talk in outcomes" rule. Non-inline
handoff inputs use an identifier link, not a raw UUID.
- Chief-of-staff persona (`AGENTS.md` for the first agent): the same
plain-language rule, so it applies in every chat.
- `skills/paperclip-converting-plans-to-tasks/SKILL.md`: task
descriptions reference other tasks and documents by identifier link,
never by raw UUID.
- No change to `skills-releases/` snapshots. They are frozen releases.
## Verification
Red on master (fix files reverted, new tests kept):
```
ui tool-row-task-reference.test.tsx
FAIL ... > falls back to a plain word while the task cannot be resolved
AssertionError: expected 'ReadtaskId: e3ecf9bc-099a-4b10-8169-9…' not to contain 'e3ecf9bc-099a-4b10-8169-9ce490a91e3d'
Tests 3 failed (3)
server user-facing-wording-guidance.test.ts
Tests 6 failed (6)
```
Green with the fix:
```
ui tool-row-task-reference.test.tsx Tests 3 passed (3)
server user-facing-wording-guidance.test.ts Tests 6 passed (6)
```
Neighbouring suites:
- `cd ui && npx vitest run src/components/task-chat
src/components/TaskChatThread src/components/transcript`: 70 files, 1448
passed, 1 skipped.
- From the repo root: `onboarding-first-task-assets`,
`paperclip-skill-utils`, `onboarding-first-task-admission`,
`hiring-operational-examples`, `cli-invocation-safety`, and the new
guidance test all pass.
- `pnpm check:token-gates`: all gates clean. The change adds no styles.
- `ui` typecheck: no errors in the changed files.
- Not run locally: `agent-skills-routes.test.ts` hit a 30 s `beforeAll`
import timeout on a cold cache. CI covers it.
## Risks
- Low risk. A row that names a task loads that task once through the
shared issue detail cache. The thread usually has it cached already. A
failed load shows "task" and does not retry.
- Rows now show other non-UUID arguments after the identifier (for
example `<identifier> · status: done`). Rows without a task reference do
not change.
- The skill text changes the guidance that agents read. They do not
change permissions, tools, or API behavior.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run in Claude Code
with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
39096a55a4 |
fix(runner): answer lease renewal outside the PRP frame chain and name lease loss (#19419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server over one authenticated PRP connection. The runner (`paperclip-runnerd`) renews its connection lease on that connection. > - The server serializes every inbound PRP frame through one promise chain per connection. Before this change, `lease_renew` waited in that chain behind earlier events. An event whose business commit stalls (for example the commit of a tool event) therefore also stalls the lease reply. The runner then stops polling its provider (ACK backpressure) and exits with "active connection lease expired" when the lease runs out. > - When that happens, the server sees only a socket close. The run log and the terminal error say "transport loss", not "lease expired". > - A follow-up from #19407 asked for renewal to be handled outside the chain and for lease loss to be named. > - This pull request answers `lease_renew` (and `pong`) as soon as the frame is decrypted. It keeps in-order decryption and the order of state-changing frames. It also records why a lease was lost and reports it as `runner_connection_lease_expired`. > - The benefit is that a slow event commit can no longer starve lease renewal, and a lease loss is visible in the run log and error code. ## Linked Issues or Issue Description Refs #19407 **What happened?** On staging, native runs with ACPX providers (Grok, Kimi, and Pi with OpenRouter/DeepSeek) ran a long foreground shell command that printed nothing (a 15–20 minute `sleep 60` loop). Each run went silent soon after `tool.execution.started`. The command finished in the sandbox, but the run never got the completion and stayed `running` until someone cancelled it. It happened in 3 of 3 attempts. The same task on Codex kept reporting activity and succeeded. **What did you expect to happen?** A quiet tool call keeps its runner connection and its lease. If the connection or lease is lost, the run log and the error code say why. **Steps to reproduce** 1. Assign a native ACPX agent a task that runs a foreground shell loop with no output for longer than 2 minutes. 2. Observe that the run goes silent and does not finish after the command ends. **Root cause analysis** - The ACPX path emits no PRP frames during a silent tool call. The runner has no timer, and the agent sends no `tool_call_update`. Codex keeps the connection busy, because its model polls the running command. Silence is the only adapter-specific factor that I found. I found no ACPX code path in runnerd that blocks the main loop: polls use a 1 ms timeout, and synchronous sidecar requests run only for server commands. - The staging build (`de1143448b`) did not include #19407. That build had no server pings. With a 60-minute connection lease (the default for `runner-prp-coordinator` and `DurablePrpCodexTransport`), the runner renews only every 30 minutes. A fully silent connection that goes through the sandbox preview proxy therefore carried no traffic at all. A route that the proxy drops without a FIN then becomes half-open forever. #19407 added 20-second pings and a 90-second termination, which closes that window. - The lease hypothesis from #19407 is real but secondary. The 60-second lease applies only when the TTL is not set. Renewal waited behind any slow event commit, and the server could not report lease loss. This pull request fixes both. ## What Changed - `DurablePrpControlPlane.attachWireConnection` decrypts each secure frame when it arrives, in counter order. Frames that arrive before the channel is secure, or behind such a frame, still decrypt in the chain. - `lease_renew` and `pong` are handled at once, outside the per-connection chain. Admission against persisted authority is unchanged. Events, command results, and handoff frames keep their serialized order. - `#renewLease` persists the renewed expiry with `save()`, so an event write that is already staged is absorbed into the same write. Before, the renewal used `commit()`, which would invalidate that staged event and close a healthy connection when renewal runs concurrently. - New `connectionLeaseFailure` getter (exported type `DurablePrpConnectionLeaseFailure`). It records `connection_lease_expired` when an authenticated connection is refused or closes with an expired lease. A new authenticated connection clears it. A credential revoked by a warm handoff is not reported, because a replacement runner owns the run. - `connectRunnerPrpIngress` reads that reason after a close. It logs `state=reconnecting failure=runner_connection_lease_expired` in the run log. If reconnect grace runs out, it fails with that code instead of `runner_ingress_unavailable` or `runner_ingress_auth_failed`, and keeps the reconnect error as the cause. - `packages/paperclip-runner/docs/durable-recovery.md` documents the new behavior. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/control-plane` (220 passed). - New: "answers lease renewal while an earlier event commit is still in flight". It was red before the fix (no `lease_renewed` while `onCommittedEvent` waits), and it is green now. It also checks that the event later commits and ACKs, and that the renewed expiry survives. - New: "keeps a staged tool-progress write when a lease renewal lands mid-write". It holds a file-backed `tool.execution.progressed` write at `fsync`. It is red with the old `commit()` renewal (the event is dropped and the connection closes), and green with `save()`. - New: "reports an expired connection lease instead of closing silently". It was red before the fix and is green now. - `pnpm --filter @paperclipai/server exec vitest run src/realtime/runner-prp-outbound.test.ts src/realtime/runner-prp-ws.test.ts src/services/native-runtime/runner-prp-coordinator.test.ts` (all passed). - New: "names an expired runner connection lease in the run log and terminal error when reconnect is refused/unauthenticated". It was red before the fix and is green now. - `src/live/runnerd-codex-transport.test.ts` has the same result with and without this change with a locally built `paperclip-runnerd`: the same 48 tests fail locally on both, for environment reasons, and the other 207 pass. ## Risks - Lease renewal now runs concurrently with an in-flight event handler. The handler builds its state candidate at commit time, and `save()` absorbs a staged event write. A new regression test covers the file-backed race. External checkpoint persistence commits events synchronously, and the renewal reply still waits for the checkpoint flush. - Renewal still stays silent while a warm handoff or a terminal lifecycle command is pending. The runner treats any other control frame inside those fences as fatal, so this pull request does not change that behavior. - No protocol, schema, or contract-digest change. The runner is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs) and a read-only exploration subagent on the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
958f9aa9f3 |
feat(issues): limit agent-created task trees to parent and child (#19312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
fb2e07ebdd |
Wait for physical authority before activating hosted warm routes (#19277)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core can give two consecutive runs, A and B, the same hosted Runner so B can continue A’s conversation. > - During that handoff, Core can record that B is next while the Runner still answers as A. > - If Core restarts at that point and tries to connect for B, the Runner rejects the connection. B’s next turn stalls. > - This PR makes Core finish the handoff over A’s connection and check which run the Runner now reports. > - Once the Runner reports B and authenticates B’s connection, B can continue on the same Runner. ## Linked Issues or Issue Description - Refs #17390 for hosted Runner reuse. - Refs #19157 for recovering a stopped Runner session before attaching the next run. ## What Changed - If the Runner still reports A, Core uses A’s connection to acknowledge the result of `run.attach`. This tells the Runner that Core saved the handoff and that it can switch to B. - Core reads the Runner’s saved state and checks it against the recorded handoff. If the Runner already reports B for that handoff, Core connects for B directly. Core waits for B’s connection to authenticate before sending its work. This applies both during an ordinary handoff and after Core restarts. ## Verification - Run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-staged-handoff.test.ts` with the native test binaries built. The regression replaces Core at four points in the handoff. Before the fix, Core tried to connect for B while the Runner still reported A. - With the fix, B and a third run each start and finish one turn. The test observes one Runner process, one controlled Codex provider process, and one conversation thread throughout. - All seven handoff cases passed. GitHub CI passed after retrying an unrelated port collision and interrupted jobs. Standard Greptile scored the commit 5/5. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42070818838) ## Risks - The host must be able to read the Runner’s state. If Core cannot verify the switch before the existing deadline, it leaves the handoff pending and sends no work for B. Live hosted verification is still separate. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning effort, through Codex with repository tools and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
2575b941d0 |
feat(task-chat): show hook and auto-review decisions as governance rows (#19266)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run on harnesses such as Codex and Claude, and the task chat shows what each run did > - Harnesses make automatic decisions: hooks block or allow tool calls, and Codex auto-review approves or denies actions > - Today Paperclip drops these decisions or shows them as generic rows. Codex `safety.review.*` always says `decision: "unknown"`, the native Rust runner drops `hook/*` and `item/autoApprovalReview/*`, and claude-agent-acp drops Claude hook messages > - An operator cannot see why an agent changed course or which action ran without a person's approval. This conflicts with Paperclip's approval-gate model > - This pull request records each hook and review decision with its reason and shows it as a small governance row in the task chat > - The benefit is that automatic decisions become auditable, and a blocked action explains itself without opening run logs ## Linked Issues or Issue Description Refs #19252 (precedent: canonical `plan.updated` events rendered as a task-chat card). Builds on #19259 (ACP `_meta` passthrough from ACPX), now merged. **Subsystem affected** Runner provider events (Rust native Codex normalizer, TS drivers, ACPX sidecar), the claude-agent-acp patch, the PRP provider-event schema, and the task chat activity UI. **Problem or motivation** Hook and auto-review outcomes reach Paperclip but are lost or flattened. Codex reviews always record an unknown decision. Native Codex runs drop hook and review notifications in the Rust normalizer. Claude hook messages never leave the adapter. Codex reuses one `run.id` for every call of a hook, and the id contains a host path. **Proposed solution** Map the real outcome, decision, action, risk and reason into additive optional fields on `paperclip.hook.v1` and `paperclip.safety.review.v1`. Forward Claude SDK hook messages. Render small grey rows: a stopped action is tinted and shows its reason, an approved review folds into the command it approved as an "auto-approved" mark, and routine hook passes stay out of activity summaries. **Alternatives considered** Separate rows for every approval. This duplicated each approved command, so approvals fold into the command row instead. ## What Changed - Runner (Rust): new `governance_events.rs`. `normalize_codex_notification` now maps `hook/started|completed` and `item/autoApprovalReview/started|completed`. Each hook call gets its own id (a digest of the run id and start time), so no host path is stored. - Runner (TS): new `governance-events.ts` with the same mapping for the TS Codex driver, and the Claude hook mapping. The ACPX sidecar publishes Claude hook updates as schema-validated `hook.*` rich events. - claude-agent-acp patch: turns on `includeHookEvents` and forwards `hook_started` / `hook_response` on `session_info_update._meta.claudeCode.hook`, with bounded output. - Protocol: additive optional fields. Hook records get `outcome`, `message`, `source` and `toolName`, and the `blocked` and `stopped` statuses. Review records get `actionKind`, `action`, `riskLevel`, `userConsent` (the reviewer's user-authorization level; keys that end in "authorization" are redacted as secrets), `rationale` and `decidedBy`, and the `timed_out` and `aborted` decisions. Regenerated the manifest and validators. - Fixed: the hook `blocking` flag read `executionMode === "blocking"`, but Codex sends `sync` / `async`. - UI: `native-run-events` keeps the full hook and review payloads. `governance-presentation.ts` writes the row labels. `fold-auto-reviews.ts` attaches an approved review to its command row. The rows show the tint, the reason line and the "auto-approved" mark. Activity summaries lead with a block or denial. - Storybook: "Task chat / Governance rows" shows every outcome, a Codex run, a Claude run and the live line. - Docs: added a governance activity section to `packages/paperclip-runner/docs/protocol-compatibility.md`. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/governance-events.test.ts src/provider-events.test.ts src/drivers/codex src/drivers/acpx src/cli`. 2305 tests pass. This includes a replay of real Codex 0.160 notifications from a session with hooks and auto-review (`test-fixtures/governance/codex-0.160-hooks-auto-review.json`, sanitized). - `cargo test --release -p paperclip-runner-core`. The full suite passes, and `governance_events::tests` replays the same fixture. - `cd ui && npx vitest run`. 11,047 tests pass. - `node scripts/typecheck-low-memory.mjs`, `pnpm check:token-gates`, and the runner contract checks (`check:protocol-manifest`, `check:protocol-types`, `check:capability-contract`, `check:protocol-coverage`, replay goldens, contract fingerprints) pass. - Live test on a local isolated instance. A Codex agent and a Claude agent each used a preconfigured home with a PreToolUse hook that blocks edits to `0001_init.sql`, plus UserPromptSubmit and PostToolUse hooks. Both runs showed "PreToolUse hook blocked a tool call" (Codex) or "PreToolUse hook blocked Bash" (Claude), with the hook's message. They also showed "UserPromptSubmit hook added context", and the group summary started with the block. - Auto-review cannot run in Paperclip today, because the native Codex runner only allows `approvalPolicy: never`. The review mapping and the approval fold are proven against real Codex 0.160 app-server output (fixture above) and in Storybook. ## Risks - Low risk to execution. These records are display-only and carry no authority. All new schema fields are optional, so older records stay valid and older UIs ignore the new fields. - The `auto-approved` fold depends on `targetItemId` matching the command's execution id. Real Codex 0.160 data confirms that it matches. When there is no match, the review stays as its own row. - In Paperclip's isolated homes, hooks do not run, because the runner never copies hooks into an isolated home. Rows appear only for agents with a preconfigured home, or with future managed hooks. - The claude-agent-acp patch changes. The ACPX profile check still passes, because the patch is Paperclip's ACPX layer and not the upstream release. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser and file editing) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
be5709a25c |
fix(runner): classify incompatible saved provider profiles (#19334)
Classify deterministic OpenCode and Cursor Cloud saved-profile mismatches with the existing permanent provider-state class. Preserve saved state and stop same-run retries after the runner refuses that state. Verified 37 native-provider backend tests, durable classified-result replay, the server policy test, rustfmt, workspace typecheck and build. Complete hosted CI passed with Greptile 5/5 and no unresolved threads. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
b1d00eeca0 |
fix(evals): derive the controlled Runner eval policy from committed defaults (#19253)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification runs real providers against 41 cases and reports them in TestOps. > - Controlled mode (#17889) adds a spend cap, private evidence storage, and cost and trace accounting. > - Operators must hand-maintain one large JSON variable for each campaign size, price review, and lockfile change. > - A one-cell policy blocked the first 41-case run, and its prices expired after 24 hours. > - This pull request derives that policy from committed defaults and the selected cases. > - The benefit is that a full run needs no policy edit, and all three safety properties stay. ## Linked Issues or Issue Description Refs #17889, #19011, #19070. **What existing behavior does this improve?** Controlled Runner protocol campaigns read `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON`. That variable holds the cell limit, a price table with a review expiry, a pinned lockfile digest, and self-hashed bound proofs. **Current behavior** A campaign with more cells than `maxCells` fails before provider work. Expired prices also fail before provider work. Any change to the lockfile needs a new pin. A one-cell policy made all 41 TestOps cases fail as infrastructure, so an operator had to edit the variable by hand before the full run. **Proposed behavior** The workflow derives the policy at preparation time. The cell count comes from the selected catalog. Prices come from a committed table with a version and a source URL. The controller installs the committed frozen lockfile. One optional variable, `RUNNER_PROTOCOL_EVAL_MAX_USD`, can override the committed $400 spend cap. **Reason and benefit** Each piece that was hand-maintained duplicated a check that already exists. The manifest digest binds every allocation. The frozen install binds dependencies to the reviewed commit. Under model-request readiness, the self-hashed bounds never changed admission or accounting. **Breaking changes** The old variable is no longer read. It can stay set or be deleted. Retained manifests that include the old proofs still replay. ## What Changed - Add `packages/paperclip-runner/scripts/controlled-evals/hosted-defaults.json`. It holds the spend cap, duration, route, Claude Sonnet 5 prices, allocation, overhead reservation and trace limits. - Add `hostedPolicy()` in `hosted.mjs`. It derives `maxCells` from the selection. It gives each campaign a validity window from preparation to one hour after its deadline. It applies the optional spend cap override. - Remove `capacityBound` and the policy self-hash from hosted preparation. `manifest.mjs` needs model `provisionalBound` only when readiness is not `model_requests`. - Workflow: stop reading the policy variable. The controller build installs the committed `pnpm-lock.yaml` with `--frozen-lockfile` and fails on any lockfile drift. Preparation checks that the runtime receipt lockfile digest equals the committed lockfile. The catalog limit is now 1 to 512 selected cells. - Keep everything else the same: the spend-cap check before provider work, private S3 retention of the manifest and derived `hosted-policy.json`, receipts, tracing, first-attempt and zero-retry gates, and paid-environment approvals. - Update `hosted.test.mjs`, `portable.test.mjs`, `runner-protocol-eval-workflow-security.test.mjs`, and `docs/runner-protocol-live-evals.md`. ## Verification - `node --test scripts/controlled-evals/hosted.test.mjs`: 8 pass. The new tests show that `maxCells` equals the selection and that the validity window covers the deadline with no committed expiry. They also show that the spend cap override applies. With the committed defaults, 41 cells validate with $339 reserved, and 49 cells fail on the $400 budget. - Negative controls still fail: drift in selection, source or attempt; a bad lock digest; missing tracing; a duration over 4 hours; a malformed or too-low spend cap; and zero cells. - `node --test scripts/runner-protocol-eval-workflow-security.test.mjs`: 20 pass. Controlled admission now passes whether the old variable is absent, still set, or malformed. A real git checkout with a modified `pnpm-lock.yaml` fails the controller guard before install. - `node --test scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs`: all pass. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - `tests/runner-e2e/workflow-security.test.ts` has 5 failures that this change does not cause. They concern `pr-trusted.yml`, `chat-catalog-testops-case.yml` and `runner-full-stack-e2e.yml`. This PR does not touch those files. - I have not run a live controlled campaign on this branch. Controlled mode runs only from the default branch. The current 41-case run uses master and the existing variable. ## Risks - The committed price table has no expiry. A provider price change needs a reviewed edit to the table. The spend cap still limits the maximum spend. - If the committed lockfile does not match the package manifests, the controller build fails before provider work. CI already requires a frozen lockfile. - The spend cap is now $400 by default. It was $339 in the last manual policy. The cap is a reservation limit, not expected spend. The expected spend for 41 cases is about $5–20. - Only ACPX Claude Sonnet 5 has a committed allocation, as in the old variable. A new controlled model needs a committed price entry and allocation. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
85e4d12856 |
fix(evals): report controlled protocol cases from safe records (#19070)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification records real provider behavior in TestOps. > - Controlled attempts keep raw records and accounting in private storage. > - A live case passed, but a later public-render command rejected its controlled context. > - The missing public report caused the final result to be stored as broken. > - This pull request creates the closed TestOps result from existing sanitized records inside the private report job. > - The benefit is correct result reporting while the privacy guard stays effective. ## Linked Issues or Issue Description Refs #18985 and #19011. The related-PR search found no separate repair for this failure. **What happened?** [The live canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378) passed all four provider checks. Its controller verified cleanup and closed its request accounting. The private canonical report recorded one passed case. The next command used `--public-viewer` while the controlled manifest remained set. The eval program rejected it with `controlled campaign evidence must remain private`. The closed exporter then had no public payload and correctly stored a broken result. **Expected behavior** Report the original case grade from sanitized structured records. Keep raw controlled records private. Keep the original checks, source bindings, and missing-evidence rules. **Steps to reproduce** Run one controlled protocol case with TestOps reporting enabled. Let the private canonical report finish. The following public-render step fails before the closed result can read the grade. **Paperclip version or commit** Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`. **Deployment mode** GitHub-hosted Linux, controlled ACPX Claude protocol qualification, first workflow attempt, zero automatic retries. ## What Changed - Add a narrow `export-runs` path to the existing TestOps helper. Bind the sanitized score and artifact to the exact attempt, case, model, provider, driver, and public-projection schema. Reuse the existing strict grade and check validation. - Keep sanitization inside the private report job. Use structured records for controlled TestOps reporting. Keep the public viewer for legacy publication. - Add record identity, privacy, grade, command, and workflow routing controls. Wire the existing helper and workflow-security files into the required Node registry. Document the reporting path. ## Verification - 43 focused campaign/sanitizer, TestOps and workflow security tests passed. The command regression writes a raw attempt, calls the production sanitizer, and runs `export-runs` on its output. It also repeats that chain after a failed-check mutation. Both new-control files now run through required `test:typescript:prep`, not only the optional publisher script. - The actual required prep ran locally with provider credentials excluded. The eval-kernel and Rust builds passed. Its Node registry executed the new producer-to-export and security controls by name: 403 passed, 2 failed, 11 existing skips. The two original package-contract failures cannot resolve this isolated clone's Codex ACP and Claude ACP package links. This is not a green local full-prep claim. The first attempt also stopped at a missing Node-types link, before an ignored clone-local link to the locked cached version restored that prerequisite. - 72 release workflow tests passed with the existing locked pnpm 9.15.4 wrapper and local loopback permission. The first sandbox run failed because loopback fixtures could not listen; the default local tool path also failed two install fixtures. No test assertions were changed to resolve these failures. - The actual repaired workflow shell ran locally against the retained canary campaign and sanitized records. It emitted one closed passed result with all four original checks. No provider request or upload occurred. - Missing or failed checks still reject a pass. Private-only markers stay absent from the closed output. Wrong model, provider, driver, attempt, case, and schemas reject. - The unmodified, pinned eval program still rejects a controlled public-viewer command. - Actionlint and `git diff --check` passed. [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886) passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates, all four Runner lanes, typecheck, build, backend and browser shards passed. Package suite: 4951 passed, 10 existing skips across 297 passed files and one existing file skip. - The required static log contains passing lines for both the production sanitizer-to-export command regression and the closed-projection security control. The prior optional-script registration did not run those tests in required CI; the one script-entry fix now enforces them. Provider/runtime and workflow bytes are unchanged by the verification-only follow-ups. - Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues. The separate Paperclip Review requested a producer-to-consumer regression. That regression and its required-CI wiring are complete. The addressed thread is resolved; no open review threads remain. - No fresh deployed reporting pass is claimed. The failed workflow and its stored broken result remain unchanged. ## Risks The exporter reads structured records inside the private job. It copies only the closed result fields and requires the existing sanitized marker and identity checks. The final reporting job still receives only those files and verifies the saved result bindings. The Python privacy guard, provider path, case assertions, budgets, deadlines, and retries are unchanged. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The inherited runtime reports a 475,000-token context window. Capabilities used: reasoning, code changes, and repository, shell, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
8002dfb06e |
fix(evals): repair bounded TestOps protocol admission (#19011)
## Thinking Path > - Paperclip lets people manage AI agent work through a governed control plane. > - Runner protocol qualification uses real providers and records selected cases in TestOps. > - A manual registration run starts an Allure watch session before it validates the case selection. > - Watch creates the job run, but it does not fill the empty job-run input in the child environment. > - The selection helper therefore stops before it can validate the new job run. > - A configured AWS concurrency ceiling can also reject a valid smaller hosted-runner request. > - Hosted tool-cache permissions can also violate Runner qualified-executable admission before a provider starts. > - This pull request captures official session identities, keeps both concurrency ceilings, and stages the same Node bytes in an owned directory. ## Linked Issues or Issue Description Refs #18985. **What happened?** The [first registration attempt](https://github.com/paperclipai/paperclip/actions/runs/41877603952) created an Allure job run, then stopped with `runner_testops_jobrun_missing`. The child inherited an empty `ALLURE_JOB_RUN_ID`. The helper made no job-run API read and executed no provider case. The current configured concurrency ceiling is 100. The controlled hosted cap is 57. The original catalog guard rejects 100 before it applies a valid requested value of 2. The [first service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41884365232) reproduced this second pre-provider failure. Its selected case was correctly reported as broken. No provider cell ran. A [second service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41887671585) passed selection, lock and cost preflight. Runner rejected a qualified ACPX launch artifact as group- or world-writable. It recorded no model request. The verified sidecar is mode 0755. Hosted Node is the only other launch artifact, which points to Node as the cause. The original run did not retain its exact mode. The TestOps reporter retained the infrastructure failure. **Expected behavior** The manual path must capture the job-run and launch IDs from the active official watch session. It must validate those IDs against the same GitHub run, job, launch and standing plan before provider work. A requested concurrency value must fit the configured ceiling and the selected runner cap. The default must use the lower ceiling. Before credentials, the worker must stage the same Node bytes in an owned directory and preserve Runner permission and digest admission. **Steps to reproduce** 1. Dispatch the trusted Runner protocol workflow with reporting enabled and an empty job-run service input. 2. Let `allurectl watch` create the registration session. 3. Observe that the original helper rejects the empty child input before selection validation. ## What Changed - Call the supported `allurectl job-run env` command for manual registration. Use a fixed argument list without a shell and bound its time and output. - Parse only numeric job-run and launch IDs. Reject missing, duplicate, malformed or conflicting values. Discard all other command output. - Bind the captured launch ID to the authoritative job-run response. Keep the service-supplied ID path and its existing checks. - Apply requested parallelism within both configured and physical ceilings. Use their minimum as the default. Keep the selected-case capacity and provider budget checks. - Snapshot the same current Linux Node bytes into a fresh owner-only directory before credentials. Verify the copy, execute it to bind its path and version, and put only that directory first on the job-local path. Record original and copied Node modes and hashes plus the actual sidecar mode and hash. Keep shared cache permissions and Runner security checks intact. - Add focused context, identity, concurrency, permission and secret-exclusion controls. Supply the production physical cap in the existing release-workflow fixture. - Fix the existing Kimi cancellation fixture to await the complete readiness sentinel within the same 100 by 10 ms bound. Keep its cancellation and cleanup assertions. ## Verification - The original empty-ID failure reproduces locally on the actual helper with all 41 standing cases and zero job-run API reads. - All 63 Node publication, projection, metrics, permission and workflow-security tests pass. The repaired manual context flows through the actual selection helper to case 194. Wrong launch, duplicate IDs, malformed values and command failures still stop selection. - The original catalog prefix rejects configured 100 / hosted 57 / requested 2. The repaired prefix admits 2, clamps a missing request to 57, keeps legacy 100, and rejects requests above either ceiling. - The full release-policy CI command passes locally: 72 release-workflow tests and 107 companion tests. The first sandbox run could not bind its loopback fixtures; the same tests passed with local-network access. The prior-head CI fixture omission is retained and fixed by supplying the same physical cap as production. - Both full static Node file groups pass locally: 418 passed and 11 existing opt-in skips, at concurrency 2. Initial runs used stale dependency links and failed. Correcting only isolated ignored links to declared cached dependencies and pnpm 9.15.4 removed those setup failures. Local Node 26.4 differs from CI Node 24; the exact-head CI below provides canonical verification. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - The new staging helper passes all four tests as a nonroot Linux user with networking disabled. An actual retained Runner comparison reproduces the qualified-artifact rejection with Node mode 0777. The same Node bytes at 0500 in a 0700 directory pass that launch boundary and reach a closed provider session. The local turn still fails without credentials or network; this is permission proof, not a protocol case pass. Local Linux Node 24.21 differs from the hosted 24.22.1 executable. - Exact final head `8e06ee5611` [repository CI](https://github.com/paperclipai/paperclip/actions/runs/41892896405) completed successfully. Typecheck, build and all four Runner lanes passed. Runner package tests passed 4950 tests in 297 files, with 10 existing skips. Fresh exact-head Greptile review is 5/5 with no actionable findings or open threads. Prior `59378f` CI Kimi empty-sentinel race and canceled chat worker remain retained. - The patched manual workflow has not run against live TestOps. A separate service-triggered probe supplies its own ID. The service probe used the original concurrency guard with a temporary configured ceiling of 57. After the probe, the original repository ceiling 100 was restored with no active campaigns. That probe does not verify this patched concurrency code. The next launch waits for the protected merge and source/lock verification. ## Risks An incompatible official CLI output fails closed. The helper never evaluates the output or records its unknown variables. The existing job, workflow URL, source, plan and provider checks remain required. An omitted concurrency request now uses the lower of the configured and physical ceilings. Explicit requests above either ceiling still fail. The worker now executes a private copy of its existing Node binary. A copy, mode, hash, path, version or sidecar mismatch stops before credential access. The copy does not make dynamically linked Homebrew Node portable; this workflow path is Linux only. This change does not alter the provider roster, policy, limits, retries or result grading. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, code editing, repository inspection and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
3a2249dd57 |
fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. **What happened?** A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. **Expected behavior** Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. **Steps to reproduce** 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. **Paperclip version or commit** Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
039e0dad2c |
Connect TestOps protocol plans to verified GitHub attempts (#18985)
## Thinking Path > - Paperclip lets people coordinate agent work through a governed control plane. > - Runner protocol evals test real provider sessions against a seeded mock control plane. > - The saved TestOps protocol plan has no GitHub executor. Its dashboard shows historical results. > - A current one-case attempt also failed before any tool call or provider output. Its retained public-safe snapshot withheld the error cause. > - This pull request connects selected cases to the existing trusted workflow and retains closed operator failure facts. > - Each selected case receives a current result. Missing evidence remains broken. Existing assertions and machine grades remain unchanged. ## Linked Issues or Issue Description **Current behavior** The saved protocol plan cannot start a GitHub qualification attempt. Historical dashboard results can be mistaken for current execution. A failed provider turn can also lose its failure category when the live snapshot applies public evidence redaction. **Proposed behavior** The existing workflow accepts TestOps service inputs, validates exact case identities before provider work, runs the selected cases, and verifies their published statuses. Eval artifacts retain only validated failure phase, category, HTTP status, and retryability. **Reason and benefit** Operators can run the standing plan from TestOps and inspect current, source-bound results. A targeted pass cannot stand in for full-plan coverage. A failed startup can be diagnosed without publishing provider messages or request identifiers. ## What Changed - Add optional TestOps selection and protected reporting jobs to the existing Runner protocol workflow. - Keep the current actor, source, provider credential, and controlled accounting gates. TestOps retains the explicitly selected controlled or legacy mode and a separate 41-case Claude roster. Controlled mode preserves its existing reviewed policy, immutable bindings, tracing and zero-retry admission. Its paid workers explicitly use GitHub-hosted runners, matching the current manifest admission contract even when the legacy AWS fleet is enabled. - Bind the exact standing plan, workflow, job run, source revisions, selected cases, and saved statuses. Accept only verified workflow URL aliases. - Create a closed Allure projection inside the report job. Transfer only fixed case identities, check verdicts, numeric usage and statuses to the reporting environment. Controlled raw/chat/controller evidence remains private. - Publish one result for every selected case. Retain missing execution/report evidence as broken. Reject unsupported passes, model mismatch and source drift. Preserve the evidence schema literal, stop publication on cancellation, and retain closed API error codes/statuses. - Tag full selections separately from targeted attempts. Add bounded ingestion observation. Apply the exact TestOps filter before the unchanged controlled provider/cell-capacity check, so a one-cell registration fits a one-cell reviewed policy. - Retain up to eight eval-only closed provider failure records before board evidence redaction. Forward the original notifications unchanged. - Add selection, reporting, privacy, and admission regression controls. Document scope and remaining acceptance. ## Verification - Direct Node protocol publication, projection, metrics, and security tests: 56 passed. Focused controls exercise both mode admissions, missing controlled policy/retries/source rejection, closed result transfer, exact metadata, and missing-result handling. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` passed. - A proposed one-cell controlled policy passed the actual hosted validator with a synthetic prepared request and zero provider calls. The actual 41-case catalog filter selected case194/get-task-context and the real hosted admission accepted it under maxCells1. Negative controls rejected the unfiltered41, unsupported provider, source, selection, origin, expiry and lock drift. This does not verify provider auth, a runtime build or TestOps publication; final merged source/lock binding is still required. - The actual diagnostic TypeScript module passed positive, negative, secret-exclusion, and unchanged-notification controls through the existing tsx runtime. A local actual-CLI startup serializer probe passed with closed facts, secret exclusion, exit status and read-only fixture cleanup. This used an isolated resolver into the existing dependency cache; canonical exact-dependency CI remains required. - Prior-head CI passed typecheck, Runner static/Rust/native-server checks and 4946 Runner package tests. One new serializer fixture failed during Linux read-only directory cleanup; this head applies the existing neighboring cleanup pattern. Unrelated chat E2E and CLI import timeouts remain retained. Latest-head required gates are pending. - Original current-source one-case attempt: https://github.com/paperclipai/paperclip/actions/runs/41860234840 . It failed with no tool calls or output. That failure remains retained and is not a passing qualification claim. - Diagnostic candidate attempt: https://github.com/paperclipai/paperclip/actions/runs/41863410602 . It retained the new closed field but still failed before output with category unknown. This disproves the claim that typed retention alone identifies the cause. - The earlier passing run https://github.com/paperclipai/paperclip/actions/runs/40627274706 used controlled Braintrust routing. Both current failures used direct legacy credentials, a different auth path. Next acceptance is one explicitly controlled case with a reviewed current policy, followed by TestOps registration and a fresh full plan. None is claimed complete by this PR body. ## Risks The TestOps contract currently binds the existing 41 cases and one recorded Claude Sonnet 5 profile. Catalog changes fail closed until reviewed. This does not qualify other providers. Admission and execution qualify only the selected accounting mode; a controlled pass does not prove direct credential health. Model-request readiness retains pending operating reconciliation. TestOps ingestion can fail independently; persisted results are checked, and missing evidence is never accepted as a pass. The new failure field contains closed facts only; raw messages, identities, credentials, and tool payloads are excluded. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, repository inspection, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the available capability details - [x] I have checked ROADMAP.md and confirmed this does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue with the enhancement template field labels - [x] I have not referenced internal or instance-local Paperclip issues - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run the available targeted local checks and reported the remaining checks - [x] I have added relevant positive and negative regression controls - [x] I have updated the authoritative protocol documentation - [x] I have considered and documented the risks - [ ] All required Paperclip CI gates are green on the latest head - [ ] Greptile is 5/5 with no open actionable feedback on the latest head - [x] I have addressed all current review feedback --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
612d9931a6 |
Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The |
||
|
Devin FoleyandPaperclip
|
4e29f00139 |
fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. **What happened?** A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. **Expected behavior** A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. **Steps to reproduce** 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. **Paperclip version or commit** `master` at `b0c4e345267`. **Deployment mode** Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - **Silent stall.** The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - **Candidate admission** carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - **The agent that loads the conversation** stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - **In-flight provider work** is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - **Launch shape.** A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - **Artifacts** are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press **Retry** once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
59da15066a |
feat(runner): deliver native Kimi Code (#18391)
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b866fa4bdc |
feat(runner): add pinned Kimi Code native runtime (#18028)
Add the pinned Kimi Code runtime, native instruction and model admission, scoped credentials, and pending-candidate recovery contracts. Retain current master release attestations and provider-pack compatibility. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
493e229d93 |
perf(runner): cache verified native executable digests (#18390)
## Thinking Path > - Paperclip manages AI agents and their task execution. > - Runner checks qualified native programs before it starts a provider. > - Installation checks and command opening can hash the same program again. > - A new sidecar process repeats this work on a later warm task. > - This pull request stores the computed digest in a private filesystem cache. > - Every cache hit still checks the independently selected trusted digest and current file metadata. > - An unchanged program can start without another full native file read. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated **qualified ACPX** native executable admission in Runner. Claude, ACPX-Codex, and Grok use this verifier in both Core-owned and plugin-backed launches. The ordinary native Codex app-server launcher is a different path and is not changed. A retained provider that opens no new native command has no saved hash work here. The measured Claude task path does not require a comment backlog. **Current behavior** Installation verification and final command opening each read and hash the native executable. The qualified Linux x64 Claude file is 241,667,256 bytes. A new verification and command lease reads 483,334,512 native bytes. Another Node process repeats both reads. **Proposed behavior** Store the actual computed SHA-256 digest by device, inode, size, nanosecond modification time, and nanosecond change time. Reuse it across Runner and sidecar processes. Compare each hit with the current build-owned digest. Hash the file on a miss or changed identity. This replaces the earlier Claude-only deferral design in this same PR. **Reason and benefit** The first verification and command lease reads the native file once. Later processes read none of its native bytes while its identity stays the same. Related: #16372 established qualified executable admission. This change is independent of other preparation PRs. It does not change provider pins or profile versions. ## What Changed - Add a shared filesystem digest cache under the absolute OS temporary root at `paperclip-runner-<uid>/executable-digests-v1`. - Use private directories and files, versioned hash-only filenames, bounded JSON entries, and atomic temporary-file replacement. Cache files contain a version and the computed digest. They contain no credentials or source paths. - Fall back to full hashing when the cache is missing, unreadable, malformed, or unusable. Keep current expected-digest selection outside the cache. - Keep cache data outside task homes. Neither `HOME` nor task cwd selects the path. The OS temporary-root selection honors absolute platform `TMPDIR`, `TMP`, or `TEMP` overrides. A relative root disables caching. Hosts must keep that root outside task workspaces. Temp cleanup causes a later rehash. - Keep the no-follow executable open, regular-file and size checks, execute permission, descriptor ownership, before/after identity checks, and preparation-to-launch identity comparison. Cache publication occurs before the final identity checks. - Use this path for standalone installation checks and final command opening. Remove the earlier pending-digest type and Claude-only launch helper. Runtime host and profile selection keep their original structure. - Test changed content, all fingerprint fields, symlinks, mode changes, inode replacement, mutation during lookup/publication, concurrent cache writers, wrong trusted pins, and cache failures. - Include a deliberate trust-boundary test: a process running as the same OS user can plant a matching cached digest for changed bytes. It opens a test descriptor and executes nothing. In plain language: Runner remembers the hash for the same file. It still checks which file it opened and which hash the build requires. The cache trusts the OS user who can write it. ## Verification From `packages/paperclip-runner`: - `../../node_modules/.bin/vitest run src/drivers/acpx/executable-digest-cache.test.ts src/drivers/acpx/installation-integrity.test.ts src/drivers/acpx/runtime-host.test.ts` — 177 passed, 6 platform-specific skips. - `../../node_modules/.bin/tsc -p tsconfig.json --noEmit` — passed. - `git diff --check` and the added-line secret, PII, private-address, and code-comment scan — passed. The offline comparison uses baseline `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` and treatment `c4356df12032448de90516ae329d8f939af10984`. Each sample starts a different Node process. Each normal sample runs the actual qualified installation verifier and opens a command lease. Seven samples per case follow one warmup. The cache is removed before each cold case. The next process reuses its entry. Baseline order alternates around the treatment pair. Command digest and package paths match in every pair. | Verifier plus command opening | Baseline | Cold cache | Later process hit | | --- | ---: | ---: | ---: | | Full native reads | 2 | 1 | 0 | | Native bytes read | 483,334,512 | 241,667,256 | 0 | | Local median, ms | 1041.26 | 602.70 | 130.17 | | Local range, ms | 994.20–1133.11 | 580.27–999.83 | 121.96–179.45 | | Median whole child process, ms | 1379.96 | 988.75 | 523.01 | Raw verifier-plus-open samples in ms: - Baseline: 1133.11, 1074.40, 1061.99, 1041.26, 994.20, 1036.12, 1037.70. - Cold: 647.13, 646.33, 999.83, 584.83, 580.27, 583.37, 602.70. - Later process hit: 179.45, 138.37, 134.39, 130.17, 121.96, 122.33, 126.68. Separate direct-opener fault probes use a disposable copy of that same native file: | Change | Native bytes read | Result | Local open, ms | | --- | ---: | --- | ---: | | First copied file | 241,667,256 | Accepted | 526.14 | | Next process, different HOME and task cwd | 0 | Accepted | 24.91 | | Changed timestamps | 241,667,256 | Accepted after hashing | 531.40 | | New inode, identical bytes | 241,667,256 | Accepted after hashing | 536.70 | | Corrupt byte, same size and exactly restored mtime | 241,667,256 | Rejected | Whole process: 841.36 | | Different independent trusted pin | 0 | Rejected | Whole process: 374.76 | The corruption probe keeps `mtimeNs` exactly equal before and after, at `1000000000000000000`. Its changed `ctimeNs` invalidates the entry. Cache expiry is not required. The changed-pin probe rejects the old actual digest without a native read. The cached image is `sha256:99ce80690a0c13328f41fd84156cc73e1ac562eaed3120fd41d04d7acdfb325d`. The Node binary is `/opt/paperclip/runner/libexec/node`, version `v24.20.0`. The artifact is Claude SDK `0.3.286`, with expected digest `sha256:fe503f65c6289d59c23e5b21ae44f03583f997dd33a2cbfc75ab4f96fb8fc73f`. The container used a read-only root, a disposable writable tmpfs, and disabled network. No provider process, model request, credential, or deployment was used. **These elapsed values are diagnostics from Linux x64 CPU emulation on an ARM64 Docker host. They are not production latency or TTFT estimates.** The exact byte counts, cross-process reuse, output equality, and invalidation results are the portable evidence. The saved phase 0 exports have no directly comparable native executable hashing span. Their broad `runner.transport.attach` spans include other work. A production comparison within 20% is therefore unavailable for this seam. The earlier local 919 ms control was also an emulation diagnostic; it is not a production baseline. The initial Docker attempt was denied access to the local socket by the sandbox. The same offline command then ran with local socket permission. An initial fault probe restored mtime through a `Date`, which did not prove exact nanosecond equality. Its raw report is retained as historical; the final probe sets and verifies an exact mtime before and after corruption. The prior Claude-only implementation and its checks and measurements remain historical evidence. They do not qualify this replacement. The superseded home-directory cache head had one browser recovery test timeout in its GitHub workflow. It expected a saved title after session recovery. The failure is retained. No browser code was changed; the current head reruns CI after moving the cache outside task homes. No full local CI suite was run, as requested. GitHub CI supplies full regression coverage. Current head `c4356df12032448de90516ae329d8f939af10984` passed the complete [GitHub workflow](https://github.com/paperclipai/paperclip/actions/runs/41145223331), including all browser shards. Its latest checks have 56 successes and 2 intended skips, including both security scans started when the PR became ready. Standard Greptile reviewed this exact head at 5/5 with no open findings. Apex then reviewed the same head at 5/5 under the documented OS-user trust policy. It reported no new findings. There are no unresolved review threads. The PR is ready for review and remains unmerged. ## Risks - This deliberately weakens protection against the same OS user. That user can write the cache and can plant a chosen digest for a file fingerprint. A cache hit trusts that record; metadata identity is not a fresh content hash. The explicit fault test documents this accepted policy. - A read-only administrator-owned Runner can use this user-writable cache. The cache has no signature, secret, or daemon. It does not require a Launcher or ROM change. - Other users cannot use normal cache permissions to write these entries. Cache errors fall back to hashing. Removing the cache directory restores full hashing. Old fingerprints leave small unused entries until the cache is removed or OS temp cleanup evicts it. Temp eviction costs one later full hash. - A changed file normally rehashes. A digest that differs from the independent current pin still fails admission. Existing command, package, credential, process-lifetime, and final dispatch checks remain. - No schema, public protocol, package dependency, profile version, or provider pin changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code editing, test execution, and tool use. This session does not expose a more precise serving model ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the details available to this session - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the enhancement template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] The targeted local tests pass; the full local suite was not run as requested - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green for the replacement head - [x] Standard Greptile is 5/5 with no open findings for the replacement head - [x] Apex is 5/5 on the current head with no unresolved findings - [x] I will address review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
8362dbff4d |
feat(runner): make every ACPX provider declare its release attestation (#18342)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native ACPX runs admit a provider only when its saved profile version and `commandDigest` match the current profile exactly. > - #18325 made every provider digest attest its upstream release, and bound the shared ACPX layer and Runner source through the pack `contractDigest`. > - The model is documented, but it lived in prose. A new provider, such as one in an open PR, could still choose another digest model without any check or guidance. > - This pull request makes the model the default path: each provider declares its attestation in one registry, and the generator, a test and a scoped rule explain the pattern where contributors meet it. > - The benefit is that new providers and upstream bumps follow one model, without a growing list of blocked legacy patterns. ## Linked Issues or Issue Description Refs #18325, #18237. **What happened?** After #18325, only Copilot, Grok and Pi went through `acpx-release-attestations.mjs`. Cursor and Antigravity had separate inline checks in the generator, and Claude and Codex had none. A new provider could add a profile with any digest. Nothing told the author how to compute it, and nothing pinned the profile versions that now change only for deliberate resets. **Expected behavior** Every ACPX provider declares how its `commandDigest` attests the upstream release. Tools and review guidance describe the pattern when a contributor adds a provider, bumps an upstream release or changes a version. **Steps to reproduce** 1. Add a new provider profile to `packages/paperclip-runner/acpx-profiles.json` with an arbitrary `commandDigest`. 2. Run `node packages/paperclip-runner/scripts/generate-acpx-profiles.mjs --check`. 3. Before this change, the check passes. ## What Changed - Add `ACPX_RELEASE_ATTESTATIONS` to `scripts/acpx-release-attestations.mjs`. Every profile declares one kind: - `installed-command` (Claude, Codex): installation hashes the upstream launch command. - `launcher` (Grok): the SHA-256 of the launcher shipped verbatim. - `closure` (Copilot, Pi): upstream versions and complete platform closure pins. - `declaration` (Cursor, Antigravity): a checked-in release declaration, as canonical JSON. - `generate-acpx-profiles.mjs` checks every profile through the registry. The separate Cursor and Antigravity digest checks are now part of the same path. No digest changes. - A profile without a declared attestation fails generation. The message says to declare one and points to the rule. - A stale digest message now says that a digest changes only with its upstream release, and that an upstream change keeps the profile version. - Add a test that pins the profile versions. It explains that a bump resets every saved session for that provider, and that a new provider starts at 1. - Add `rules/acpx-profile-identity.mdc`. It describes the identity model, how to add a provider, when to bump a version, and review criteria. `AGENTS.md` and `.greptile/files.json` route the profile, attestation, contract-registry, ACPX driver and provider paths to it. - Update the profile identity section in `packages/paperclip-runner/docs/durable-recovery.md` with the attestation kinds and the version policy. ## Verification - In `packages/paperclip-runner`: - `node scripts/generate-acpx-profiles.mjs --check` passes. Running it without `--check` changes no digest. - `node --test scripts/acpx-release-attestations.test.mjs scripts/contracts/contract-fingerprints.test.mjs`: 22 tests pass. New cases: every profile declares an attestation, a new provider without one fails with guidance, and Cursor and Antigravity declaration changes affect only their own digests. - `vitest run src/drivers/acpx/profile-compatibility.test.ts`: 15 tests pass, including the version table. - `tsc -p tsconfig.json --noEmit` passes. ## Risks - Low risk. No digest, version or runtime admission behavior changes. - A new provider PR must add one registry entry. The generator error states what to add. - The version table needs an edit for each deliberate reset. Upstream bumps do not touch it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
abc270ee4c |
fix(runner): attest upstream releases and bind ACPX uniformly for every harness (#18325)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native ACPX runs admit a provider only when its saved profile and `commandDigest` match the current profile exactly. > - For Copilot, Grok and Pi, `commandDigest` hashed dozens of Paperclip source files. Claude, Codex, Cursor and Antigravity hash only their upstream release. > - Any change to a bound source therefore advanced these profiles. #18212 alone moved Copilot from v88 to v91 and Grok from v15 to v19. Each bump reset in-flight sessions and added immutable identity fixtures (about 160 on master). > - #18237 added the provider pack `contractDigest`, which now guards Runner source compatibility at the pack boundary. > - This pull request changes the Copilot, Grok and Pi digests to upstream release attestations, as the other providers already use. > - It also binds the shared ACPX layer the same way for every ACPX harness: the profile's `acpxVersion` and the pack `contractDigest`, which now covers the Runner's ACPX patch. > - The benefit is that Runner source changes no longer reset these sessions or need new profile versions and fixtures. ## Linked Issues or Issue Description Refs #18237, #18212, #17972. **What happened?** A Runner change that touched a bound source file changed the Copilot, Grok or Pi `commandDigest`. The author had to advance the profile version, regenerate an identity fixture and keep every old fixture. On deploy, saved sessions for that provider failed the exact admission check and started fresh. **Expected behavior** A provider's `commandDigest` changes only when its upstream release changes: an upstream version, a platform closure, the ACPX patch or the Grok launcher. Runner source compatibility is the job of the pack `contractDigest`. **Steps to reproduce** 1. On master, change a comment in `src/drivers/acpx/runtime-host.ts`. 2. Run `node packages/paperclip-runner/scripts/grok-profile-identity.mjs --check`. 3. The check fails. The Grok profile must advance, and the next deploy resets Grok sessions. ## What Changed - Add `packages/paperclip-runner/scripts/acpx-release-attestations.mjs`. It computes the release attestation for each provider: - Copilot and Pi: a canonical-JSON hash of the profile version, upstream package versions and the platform closure pins (Pi also binds its Node version). - Grok: the SHA-256 of `launcher.cjs`. Admission already hashes these bytes. - The attestation binds no Paperclip source, shared ACPX layer, absolute path, pack manifest digest or install layout. - Bind the shared ACPX layer once for every ACPX harness (Claude, Codex, Copilot, Cursor, Grok, Pi, Antigravity): - Add the Runner's ACPX patch (`patches/acpx@<runner acpx version>.patch`) to the pack `contractDigest`. A patch change now retires older image packs for every harness. Other packages' ACPX patches are not read. A build context without the Runner's patch fails. - Remove the ACPX patch hash from Cursor's release declaration (`cursor-contract.json`) and advance Cursor once to profile 17. Copilot, Pi and Cursor no longer bind ACPX in their own digests, like Claude, Codex, Grok and Antigravity. - Saved inputs for every harness already must match the current `acpxVersion`. - Claude and Codex digests do not change. - `generate-acpx-profiles.mjs` writes these digests to `acpx-profiles.json`. `check:acpx-profiles` refuses a stale digest. - Move the Copilot closure pins to `copilot-closure-pins.ts`, so that the generator can import them. `copilot-installation.ts` re-exports them. - Advance each profile once and then freeze it: Copilot 92, Grok 20, Pi 37. A version change at the switch keeps the rollout on the same path as every earlier profile bump. - Replace the historical version lists in `profile-compatibility.ts` with a range. Every revision from 1 through the current one decodes. Claude and Codex still decode revisions 1 through 5. Launch and recovery admission still require the exact current profile and digest. - Delete: - `scripts/copilot-profile-identity.mjs`, `scripts/grok-profile-identity.mjs` and its test - the Grok launcher source stamp - 159 Copilot, Grok and Pi identity fixtures, including three unreferenced `copilot-mainline-v*` fixtures. The Cursor fixtures stay. - Rewrite the affected tests as behaviour checks: - Each provider admits only the exact current profile. Every earlier version decodes and is refused at admission. - The last source-bound digests, and a changed digest at the current version, are refused by recovery and by the backend factory. - The Rust Pi descriptor test uses the generated profile instead of fixtures. - New `acpx-release-attestations.test.mjs`: a relocated copy of the inputs gives the same digests, each upstream input changes only its own providers, and the generator refreshes and checks the manifest. - Remove the fixture exceptions from `.dockerignore`. Add the attestation script and the generator to the Daytona image content inputs. - Update the Pi install probe and the Rust integration constant for the new Pi digest. - Update the docs to describe the current attestation model: `doc/grok-native-runner.md`, `doc/RUNNER-INTEGRATION-PLAYBOOK.md`, `doc/observability.md`, `packages/paperclip-runner/docs/durable-recovery.md`, and the Copilot, Pi and rich ACP architecture reports. Each report now opens with the current profile identity; earlier source-bound profiles remain as dated chronology. No doc references a deleted file. ## Verification - In `packages/paperclip-runner`: - `node --test scripts/acpx-release-attestations.test.mjs scripts/contracts/contract-fingerprints.test.mjs`: 19 tests pass, including a Runner ACPX patch edit that changes the contract digest, other patch edits (including `acpx@0.12.0` for `adapter-utils`) that do not, and a build context without the patch that fails. - `node scripts/generate-acpx-profiles.mjs --check` passes. - `vitest run src/drivers/acpx src/backends src/contracts`: 1647 tests pass, 9 skipped. - `tsc -p tsconfig.json --noEmit` passes. - From the repository root, `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/daytona-image.test.ts`: 11 tests pass. - In `server`, the 8 test files that use ACPX profiles, pack digests or `contractDigest`: 1617 tests pass. - Not run locally: the Rust tests and `cargo fmt`, because this machine has no Rust toolchain. CI runs them. - Before this change, a synthetic upgrade check built two Runner revisions that differ only in shared source. A Claude checkpoint saved under the first revision rebinds under the second, and a changed Claude `commandDigest` starts fresh. This confirms that the provider JSON has no other build-dependent field. ## Risks - **One reset at rollout.** The Copilot, Grok, Pi and Cursor digests and versions change once. Saved sessions for these providers start fresh on their next run, as after any earlier profile bump. After this change, Runner source changes no longer cause this reset. - **Lost "source changed, so requalify" signal.** The old digest forced a new profile version when bound source changed. Nothing enforced requalification: Copilot was last qualified at v47. Pack-boundary contract changes are now caught by the pack `contractDigest`. - **Image packs are retired once for every provider.** Adding the ACPX patch changes the pack `contractDigest`, so the controller uploads its bundled pack (about 16 s per remote start) until images are refreshed. Claude and Codex sessions are not reset: their digests and versions do not change. - **Grok launcher and closure pins.** The Grok digest covers only the launcher bytes. Grok's runtime version is still checked separately at admission, as before. - **Grok digest equals early Grok profiles.** The launcher bytes have not changed since profile 1, so the new Grok digest equals the published profile 1 digest. The profile version still separates them: launch, recovery and the backend factory require profile 20. An image pack from that period has no `contractDigest`, so the pack gate refuses it. `remote-provider-pack-verification.test.ts` now covers this case. - **Docker context checks** run `generate-acpx-profiles.mjs --check`, which now imports `.ts` pin files through Node type stripping. `materialize-pi-distribution.mjs` already relies on this on Node 24. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
scotttongandPaperclip
|
71f0c0c99e |
feat: add opt-in Coordinator workspace for chat, tasks, and decisions (#17983)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/147/changes/paperclip/147 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People coordinate work through conversations, tasks, and requests for input. > - These surfaces must share task state and preserve the context of each decision. > - The existing workspace makes people switch surfaces to follow that context. > - This pull request adds an optional Coordinator workspace with linked conversations, tasks, documents, and decisions. > - Each person can enable it in Settings > Experimental and return to the classic workspace. > - The benefit is a coherent path from a request to an assigned task and reviewed work. ## Linked Issues or Issue Description **Subsystem affected** UI, server, shared contracts, and database. **Problem or motivation** People need a consistent way to start tasks from agent conversations, answer questions, review work, and return to the source context. Decisions and task state must agree across these surfaces. **Proposed solution** Add a personal, default-off workspace experiment. Keep one conversation per agent. Add task proposals, compact task and decision cards, contextual document panels, and a Needs You queue with search, filters, sorting, grouping, and history. **Alternatives considered** Replacing the classic workspace for everyone would remove the opt-out path. Keeping separate decision state in each surface would create conflicting answers. This change uses the existing canonical task and interaction records instead. **Roadmap alignment** This supports the existing agent chat, task management, and human review capabilities. It is an experimental presentation choice. The user requested this isolated implementation after design review. No Discord discussion is claimed. ## What Changed - Add per-person workspace preferences, a default-off setting, and an additive migration. - Add Coordinator navigation, pinned destinations, task lists, agent switching, and contextual panels. - Link task suggestions and creation to the source conversation. Default new task ownership to the current agent unless the user specifies another owner. - Share pending questions, decision receipts, exact review targets, and history across tasks, chat, and Needs You. - Preserve task access, assignment authorization, revision binding, and required output review gates. - Add compact document links, relative timestamps, composer controls, and reduced-motion support for task and decision transitions. - Add component stories, regression coverage, and capability-map documentation. ## Verification - Final head: `4b839aae1e314bd7e8635a766f1d205ff1c03ba6`, based on master `8bc051245f77a47bee0f8ba45896b8bfa5922e0a`. Verified October 29, 2026 at 00:10 UTC: **MERGEABLE, 56 checks passed, four conditional skips, no pending or failed checks**. [Current CI run](https://github.com/paperclipai/paperclip/actions/runs/40790639688) passed without a rerun on this head. - Fresh Greptile **5/5**, zero new comments and zero open review threads; Superagent passed with no suspicious changes. [Checks](https://github.com/paperclipai/paperclip/pull/17983/checks). - Local workspace typecheck/build passed after the latest rebase. The final account-read patch passed UI typecheck/build and all 12 related preference/settings tests, including a delayed real-client request across account changes. Earlier scoped checks cover review deduplication/replay, decision deep links, cross-surface state, access boundaries, and migration replay. - Local database/browser rechecks after the latest rebase were limited by exhausted macOS shared-memory IDs: `initdb` failed before database creation. The local run passed 31 UI/snapshot tests and skipped 24 database tests; those skips and the blocked local browser start are not claimed as passes. The independent Linux CI database and browser checks passed. No running dev database or host setting was changed. - Upstream migration 0403 is preserved. Preferences and durable review retry aliases are generated as replay-safe migrations `0404_tense_silver_sable.sql` and `0405_harsh_avengers.sql`; five newest snapshots are retained. Every detected conflict was rebased and pushed before waiting for CI/review, as requested. - The PR is unmerged. The active master ruleset still requires code-owner approval for the changed Paperclip skill; no approval or rule change was performed. <details> <summary>Earlier verification failures, integration history, and evidence</summary> These entries preserve prior checkpoints and failures; the current acceptance result is above. - Focused UI and server checks from the isolated build are recorded in the implementation plan. - Browser checks covered the opt-in shell, navigation, task/document links, and decision card states. No new live provider trial is claimed for this handoff. - Current acceptance head: `4b839aae1e314bd7e8635a766f1d205ff1c03ba6`, rebased onto master `8bc051245` immediately after its GitHub review-repair migration took 0403. Preferences are generated as idempotent `0404_tense_silver_sable.sql`, preserving upstream SQL/journal/snapshot history and five newest snapshots. GitHub reports MERGEABLE after the immediate lease-protected push. Scott's updated instruction is to rebase and push every conflict resolution first, without waiting for CI or Greptile; verification follows the push. - The final reviewed feature tree was consolidated for repeatable single-commit rebases. The complete previous commit history remains reachable through local recovery ref `codex/coordinator-recovery-before-rebase-20261028` (resolved snapshot `0e1bbdf4e`). The actual rebase completed, and its tree hash exactly matches that resolved snapshot. The force-with-lease required the known remote head `0e6ef7c44`, protecting against concurrent branch changes. No PR merge or permission/runner setting change occurred. Fresh CI/review are in progress for the current head. - `2001132fa` received Greptile 5/5 with one nonblocking account-switch read finding. Current head detaches path-coalesced preference GETs and disables HTTP caching, retaining React Query's account-scoped sharing. A delayed real-client GET regression proves account B can load its saved choice before account A responds, and A's late response only updates A's cache. Preference/settings tests 12/12 and UI typecheck passed; its review thread is resolved. Fresh current-head CI/reviews are pending. The prior CI chat shard 3 ended with “The operation was canceled” and no assertion failure ([job 123171595631](https://github.com/paperclipai/paperclip/actions/runs/40789688333/job/123171595631), `/tmp/pr17983-2001132-chat3.log`); no code/timeout change is attributed to that cancellation. - `843e823b4` reached fresh Greptile 5/5 with zero comments/open threads and Superagent pass. All browser tests including the repaired YAML shard passed, and 54 checks passed. General server shard 9 ended with “The operation was canceled” and no failed assertion; its test step was marked cancelled (job [123164350893](https://github.com/paperclipai/paperclip/actions/runs/40787501761/job/123164350893), log `/tmp/pr17983-843e-server9.log`). A single unchanged-head rerun was started after the workflow completed, but master moved while it ran. A local nested-clone check timed out in its 30-second database-startup hook and then failed cleanup before the 20 database cases ran; this is retained as a local verification limit (`/tmp/pr17983-843e-nested-clone-check.log`), not claimed as a pass or proof of the CI cancellation cause. - Following Scott's instruction, the new conflict was rebased and pushed immediately without waiting for that rerun. Upstream's `0403_natural_cerise.sql` is preserved; regenerated preferences and retry-alias migrations are `0404_tense_silver_sable.sql` and `0405_harsh_avengers.sql`, replay-safe with five newest snapshots retained. The YAML helper conflict combines upstream's click and header behavior with the editor-selection guard and injected regression. Recovery ref `codex/coordinator-recovery-843e823b4` preserves pre-rebase work. Current head is MERGEABLE. Full workspace typecheck/build passed; UI and snapshot checks passed 31 tests, but 24 database tests were skipped by the local support probe, and the local browser server could not initialize PostgreSQL. A direct disposable `initdb` diagnostic established the host limitation: `could not create shared memory segment: No space left on device` (`shmget`, shared-memory IDs exhausted, not disk space). No dev database or host setting was changed. Diagnostics: `/tmp/pr17983-2001132-initdb-probe.log`, `/tmp/pr17983-2001132-db-probe.log`; checks: `/tmp/pr17983-2001132-tests.log`, `/tmp/pr17983-2001132-browser.log`, `/tmp/pr17983-2001132-types.log`, `/tmp/pr17983-2001132-build.log`. These local skipped/blocked checks are not claimed as passes; fresh CI on independent runners is in progress. - `d18f683cd` reached fresh Greptile 5/5 with zero comments/open threads and Superagent pass, but [E2E shard 7](https://github.com/paperclipai/paperclip/actions/runs/40785966243/job/123159719925) failed parsing an Advanced YAML clipboard read beginning with “Skip to Main Content.” The log is retained at `/tmp/pr17983-d18f-e2e7.log`. Current head hardens the test helper to verify editor focus and selection, clear stale clipboard content, and require the YAML header at the start of a fresh copy before parsing. Saved-state assertions and timeouts are unchanged. All six Advanced YAML browser journeys passed on a disposable local instance (41.0s); the exact lost-response journey also passed with a one-time injected surrounding-page copy, verifying rejection/reacquisition (17.6s). The precise browser focus/clipboard timing in CI remains unproven; the observed invalid-read path is now guarded and deterministically exercised. Logs: `/tmp/pr17983-yaml-clipboard-scope-browser.log`, `/tmp/pr17983-yaml-clipboard-injected-browser.log`. Fresh current-head CI and reviews are pending. - `19e6dcb0b` passed Superagent and Greptile 5/5, with one nonblocking deep-link finding. Current head preserves attention links using `?decisionId=`: the requested pending decision is rendered even beyond the first 40 cards, focused and scrolled into view, then only that parameter is consumed so later refreshes do not refocus. Other URL parameters remain. Needs You tests 10/10 and UI typecheck passed; its review thread is resolved. The preceding post-rebase regression run passed 45/45 tests. Fresh current-head CI/reviews are pending. - `6d772b7cc` reached Greptile 5/5 with zero comments/open threads and Superagent pass. While 32 checks had passed, master moved and caused a documentation conflict in `maps/feature-map/onboarding.md`. Rebased immediately, preserving both onboarding paragraphs. The complete four-commit branch was pushed as `19e6dcb0b` before waiting on CI/review, per Scott's instruction. A transient Git index lock interrupted the first continue; the command sequence briefly pushed intermediate `34e298c07`, then the completed rebase was immediately pushed. All feature and security/retry fixes are present; local recovery ref `codex/coordinator-recovery-6d772b7cc` preserves the pre-rebase history. The only tree differences from that ref are upstream's two smoke-test files and the combined documentation paragraph. GitHub reports MERGEABLE. Fresh current-head verification is in progress; prior-head review results are not current acceptance. - `21efa3120` passed Superagent but Greptile scored 4/5 for a valid retry-key persistence bug: an acknowledged duplicate key could reopen a resolved review. Current head adds an additive, replay-safe `0404_light_swarm.sql` migration for indexed, task-scoped aliases from acknowledged retry keys to the canonical interaction. Alias registration and retry lookup occur under the existing task lock; payload/actor mismatch still conflicts, and each canonical review permits at most 100 aliases. Original and acknowledged keys replay the terminal result rather than creating a new gate. Five newest migration snapshots are retained. Service/routes tests 165/165 passed; final quality review plus migration replay/snapshot checks 25/25 passed; server typecheck passed. The P1 thread is resolved in code; fresh current-head reviews and CI are pending. - `69c2c7543` passed 55 checks and Greptile 5/5 with no open Greptile findings, but Superagent check [123140675812](https://github.com/paperclipai/paperclip/pull/17983/checks?check_run_id=123140675812) correctly flagged unbounded pending quality-card creation. Current head fixes that under the existing task-row lock: equivalent requests with fresh idempotency keys reuse the pending card; conflicting requests for an unresolved deliverable return the existing interaction ID; pending cards are capped at ten per task across all reviewers. Distinct outputs remain independent, and already-answered items in a partially pending batch can receive a fresh review. Added real-database concurrency, cross-card duplicate, partial-answer, and hard-cap regressions. Quality review 22/22 tests and final server typecheck passed; earlier service/routes 163/163 tests passed before the final partial-answer refinement. One added assertion initially expected a service HTTP error, but duplicate keys are rejected earlier by the existing shared schema; the assertion now verifies that validation. Fresh current-head CI/reviews are pending; the action-required security result is not treated as passed. - `c9c575e5f` completed all 56 checks successfully with four conditional skips, Greptile 5/5, and Superagent pass. Its two P2 review findings are fixed in the current head: independently bound quality approvals now each remain visible in attention; changing/clearing URL-backed assignee/status filters updates the URL and survives task navigation. Both review threads are resolved. Regression evidence: attention service 34/34 tests passed, CoordinatorTasks 20/20 passed, UI and server typechecks and the full workspace build passed on `69c2c7543`. The first new UI test run caught an assertion racing React Router's transition; waiting for the navigation before asserting passes without weakening the navigation/remount checks. These older-head green checks are not claimed for the new fix commit. - Preceding `0e6ef7c44` completed all 56 checks successfully with four conditional skips, fresh Greptile 5/5 and no open findings, and passed Superagent. A fifth migration-metadata collision arrived before the final Runner job finished. Those green results are preserved as prior-head evidence, not current acceptance. - Previous fully verified migration head: `0e6ef7c44696a377f35e69521b93587576374bd0`, integrating master `f8226b051` after the fourth migration-number collision. Master added terminal GitHub-response migration 0401 while the final native check ran; upstream 0401 SQL/journal/snapshot is preserved unchanged, and preferences are generated as idempotent `0402_yielding_blink.sql` on top. Exactly the five newest snapshots remain. GitHub reports MERGEABLE after the push. All 1,155 relevant migration/review/interaction/native-execution/remote-assets tests, workspace typecheck/build, migration guards and whitespace checks pass. Fresh exact-head CI/Greptile/Superagent are pending. - Immediately preceding head `36c67cf03` completed all 56 checks successfully with four conditional skips, fresh Greptile 5/5 and no open findings, plus passed Superagent security. No retry was needed for that head's CI. Master advanced and introduced only a migration-metadata conflict while its final native check ran, so that green result does not establish acceptance of the current integration. Four consecutive collisions (0398, 0399, 0400, 0401) during verification are retained as concrete integration churn; a brief pause of competing migration merges was requested, without assuming it happened or changing any repo setting. The PR has not been merged. - Prior fully verified head: `36c67cf03`, on the clean `30059c0ed` integration of master `efd65c112`. The last Greptile P2 is fixed: Needs attention retains the participant-agent scope through authoritative server membership queries for each pending task, including historical participation when another agent currently owns the task. It does not approximate participation with current assignment or enumerate every company task. The actual view-switch regression and all 31 focused task/routing/collection tests, UI typecheck/build, token gates and whitespace checks pass. Greptile completed a fresh 5/5 review on this exact head (review 19) with zero new comments and no open review threads. CI and Superagent remain in progress; no current-head CI failure has been reported. - Head `30059c0ed` received Greptile 5/5 with that one nonblocking participant finding, now fixed. Superagent passed that head. Current-base integration restored the ephemeral RunsOn fleet after a superseded workflow retained queued aggregate jobs: normal cancellation was insufficient; the single force-cancel request was interrupted but readback confirmed that obsolete run cancelled and current CI began on the trusted fleet. No cancellation retry or runner/security/workflow changes were made. Its partial CI was superseded by the necessary participant fix and is not current acceptance. - Previous compatibility/base-sync head: `30059c0ededc0eab4e04e8299a884f36db8b292f`, integrating master `efd65c112` and the Inbox/workspace compatibility fixes. GitHub reports MERGEABLE after the push. Coordinator keeps Inbox standalone under More without rewriting the instance's Combined Inbox preference; opting out restores that preference. Saved merged-Inbox URLs resolve to their original owner, task status presets remain usable, and canonical repeated `workspace` parameters retain scoped, deduplicated pagination (including Needs attention). All 192 broader routing/Inbox/sidebar/privacy/review tests and the final 16 task-page tests pass, as do workspace typecheck/build, token gates and whitespace checks. The two review threads are resolved. Fresh exact-head CI, Greptile and Superagent results are pending. - The preceding `d5b9be5bd` remained mergeable but its CI routing gate intentionally fell back to GitHub-hosted runners: “stacked base merge commit does not contain the current base branch.” All 45 downstream jobs stayed queued. Current master was merged cleanly to refresh the pinned base; no runner/security settings or workflow logic were changed. Greptile scored that preceding head 4/5 with the two compatibility findings fixed here. Its automatic Superagent scan was inconclusive at checkout; the documented retry was still pending when superseded. Those results do not establish acceptance of this current head. - Previous approval/migration head: `d5b9be5bd3b8d6b0437e42bf48138377f34fa67d`, integrating master `ba5bba86d` and the stale-output approval correction. GitHub reports MERGEABLE after the push. The fresh P1 finding is fixed: every submitted item now validates and locks its bound output inside the resolution transaction, whether or not it carries a required-quality-review binding. Real database regressions verify that an advanced document revision or deleted attachment refuses the entire batch without recording approvals, and current ordinary output reviews still resolve. All 272 combined migration/review/interaction tests pass; workspace typecheck/build passed. Fresh exact-head Greptile and CI are pending. The automatic Superagent scan ended inconclusive at checkout; one documented on-demand retry is pending. These are explicit verification limits, not passed security results. The P1 thread is resolved. - Master consumed the next migration slot three times during this verification pass: confirmation recovery took 0398, continuation recovery took 0399, and GitHub acknowledgement recovery took 0400. Each upstream SQL/journal/snapshot is preserved unchanged. Preferences are now generated as idempotent `0401_white_lenny_balinger.sql` from upstream 0400; exactly five snapshots remain, and replay coverage locates this migration by table content. This is recorded concurrent integration churn, not a claim that older green results apply to the latest base. - Head `ed3b68da9` fixed the two history/URL findings with 26 focused regression tests, server/UI typechecks, workspace build and token gates passing. Greptile then scored that head 4/5 with the stale-output P1 corrected in the current head. Its partial CI is superseded, not counted as current acceptance. - Previous filter-fix head: `ed3b68da9`, based on the `8458d39d9` merge of master `063cfd438`. The two fresh Greptile findings are fixed: history outcome filters now match recorded per-item verdicts in SQL before pagination (mixed reviews remain in both applicable outcomes), and same-page URL assignee changes/back/forward navigation update that filter without clearing independent filters. All 26 focused database/navigation/collection tests, server/UI typechecks, workspace build, token gates and whitespace checks pass. The two review threads are resolved. Fresh exact-head CI, Greptile and Superagent results are pending. - The `8458d39d9` migration integration also passed all 328 targeted migration/interaction/confirmation tests, workspace typecheck/build and Superagent security scan. Greptile scored that head 4/5 with the two functional gaps fixed in the current head; those prior-head checks are not current acceptance. - Migration integration head: `8458d39d9721f4cb312c2390070a0609f8cddd1a`, integrating master `063cfd43813cabf18417605fb9eefb9aacf95c8d`. GitHub reports MERGEABLE after the push. Master advanced twice during this conflict-resolution pass: first its confirmation-recovery migration took 0398, then its continuation-recovery migration took 0399. Both upstream migrations and journal entries remain unchanged; Coordinator preferences are now generated as idempotent `0400_sad_guardian.sql` from upstream's latest snapshot. Exactly the five newest snapshots remain. The replay regression now locates the table migration by content instead of hardcoding a renumbered filename. - All 328 targeted question/review/confirmation/migration tests passed again after the second merge. Workspace typecheck/build and fresh exact-head CI/reviews are in progress. The preceding `0c46f46d6` passed workspace typecheck/build and Superagent security scan, but was superseded because master consumed migration 0399 while those checks ran. That prior-head result is not acceptance of the current head. No production settings or runner/security permissions were changed, and the PR has not been merged. - Previous integration head: `0c46f46d6f2b23417ab872ce3cb6ff2673647287`, integrating master `287b04d72f1b9fca1412e05435c75b8302ba0710`. GitHub reports MERGEABLE after the push. The renewed conflicts were `0398_snapshot.json`, `_journal.json`, and the top imports in `issue-thread-interactions.ts`. Upstream's `0398` confirmation-recovery index and journal history are preserved unchanged; Coordinator preferences were regenerated as idempotent `0399_bouncy_miek.sql`, with replay coverage updated and exactly the five newest snapshots retained. Both terminal confirmation notifications and Coordinator task assignment/required review behavior are preserved. - All 328 targeted interaction/review/confirmation/migration tests pass, as do workspace typecheck, map validation and whitespace checks. The workspace build and fresh exact-head CI/Greptile/Superagent checks are pending. The first migration-generation attempt hit the sandbox's local IPC restriction before generating files; the unchanged authorized local generation completed successfully. Prior-head green checks below do not establish acceptance of this merge. - Previous acceptance head: `8b7dcf739ed15afc51351fdfa0e0e5fbc75ab788`, with master `1aa3a21ac` integrated and GitHub reporting MERGEABLE. Greptile completed a fresh 5/5 review on this exact head with zero new actionable findings and no open review threads. The [fresh CI workflow](https://github.com/paperclipai/paperclip/actions/runs/40752864319) completed SUCCESS on attempt 2 at 19:40:08 UTC, including all eight E2E shards. Final completed check set: 55 succeeded, four conditional checks skipped, zero pending/queued/running checks, and one neutral/inconclusive Superagent scan. The auxiliary commitperclip review and isolated native Runner compilation/cache-reuse check both passed. This is not a claim that Superagent produced a passed security verdict. GitHub still reports MERGEABLE against current master `60b8089a44b4eaa37e627adbfdf2f130bda13335`. - Current-head E2E: all eight shards and the aggregate passed, including the previously failing YAML browser journey. Serialized server shard 9/9 failed: the protected YAML projection test exceeded its original 15-second timeout; its late update request then reached the next test's reset mock (the captured payload is the prior test's model/label update). All 88 tests in that exact unchanged file passed locally in 38.03 seconds, with no timeout or assertion changes. The original slow-request cause remains unproven. CI failure log: `/tmp/pr17983-8b7-serialized9.log`; local recovery: `/tmp/pr17983-adapter-yaml-repro.log`. The single bounded unchanged shard rerun passed, and the workflow completed SUCCESS. Rerun log: `/tmp/pr17983-8b7-serialized9-rerun.log`. The initial failure is retained; a passing retry does not establish its original slow-request cause. - Greptile scored preceding head `4873704be` 5/5 with one valid P2: opened task decision records missed company-scoped live invalidation. The current head uses the existing company decision prefix for resolution and effect-completion events, and resets private record payloads on access changes. Its thread is resolved. All 107 focused record/live-update tests, UI typecheck/build, token checks and whitespace checks pass. - CI on `4873704be` failed E2E shard 7/8 in the Advanced YAML lost-creation-response journey: its clipboard helper selected page text starting with “Skip to Main Content” instead of YAML. The exact unchanged journey passed in a disposable local browser instance (1 test, 17.1 seconds). No assertion, timeout or production behavior was weakened; the original failure cause remains unproven. Failure log: `/tmp/pr17983-487-e2e7.log`; local check: `/tmp/pr17983-agent-yaml-repro.log`. The full unchanged Advanced YAML browser suite also passed locally (6 tests, 37.8 seconds). Fresh current-head CI subsequently passed this journey and all other E2E shards. - Both the automatic and one supported on-demand Superagent scan on `4873704be` ended neutral/inconclusive: “Pinned PR scan merge-base checkout failed or timed out.” These do not count as a passed security verdict. The current head's automatic scan and one documented on-demand retry also returned the identical neutral checkout failure, with zero annotations or further diagnostics. This is an external scanner blocker, not a passed security verdict. No security settings or permissions were changed. - Recorded decision links now render and focus the full source-task record in both Classic and Coordinator, including recorded choice, submitted inputs, and the existing effect-result resolver. Fresh terminal records take precedence over older open-decision cache entries. Context and account checks keep mismatched or inaccessible records out of the DOM; no decision-history tab was added to tasks or chat. The shared company live-run feed now drives live-only filtering, including ended runs and terminal tasks with stale run entries. All 213 focused tests, UI typecheck/build, token and whitespace checks pass; the final focus follow-up passed all 188 task-detail tests and UI typecheck. The first new navigation test failed because its test host omitted company context; the host was corrected before the passing run. - Previous integration head: `375c5e89516705050d94bab99336212841b2e6e1` merges master `1aa3a21ac` after four additional upstream commits. The sole conflict was competing imports in `server/src/services/issues.ts`; both quality-review policy and inbound wake-notice notifications are preserved. The PR-specific review import now sits beside execution-policy imports instead of the frequently changing first line. GitHub mergeability is rechecked after the push. - On the combined master tree, workspace typecheck/build, all 153 affected UI tests, map validation, and whitespace checks pass. Four server suites passed 77 tests; the fifth suite's module-import hook timed out while broad verification was running. Its unchanged isolated run passed all 41 tests in 7.2 seconds with the original timeout. This records recovery without claiming the original import-timeout cause is proven. CI on `375c5e895` passed every substantive job, including all eight E2E shards and Superagent; Greptile found the two functional gaps fixed in the current head. A short-lived `669d90876` commit added the main fixes before the focused navigation follow-up. - Immediately preceding head `9d9737b49` completed all 56 checks successfully with four conditional skips, Greptile 5/5 and no open findings, plus a successful Superagent scan. The signoff E2E shard passed on its single accepted rerun; its initial cancellation/timing failure is retained below. Master moved again while those checks ran, requiring this final integration rather than treating the prior head's green result as current acceptance. - Previous head `9d9737b496911a6054d0ea22d1b42b28b115e9ab` includes master `d30c8f42b842327ee1c18c41a2efd6ba623da5cd`; that head was subsequently conflicted by moving master. The master merge preserved durable account-bound writes and Coordinator UX. Workspace typecheck/build, 416 focused UI tests, and 142 affected comment-route tests passed. A stable full UI run on preceding head `938ad1665` passed all 9,833 tests across 836 files; its final full workspace build also passed. - Greptile found nested proposal project inheritance and Classic bound-revision links on `91be74952`. Both were fixed with 142 interaction-route tests, 113 focused UI tests, and UI/server typechecks. Its next review identified a valid draft-preservation bug in the new Classic loading UI. Head `9d9737b49` keeps loading/error notices outside editors and annotations while preserving dirty drafts; 115 focused review UI tests pass, including pending/failed revision loads and continued draft editing. All related threads are resolved. Greptile completed a fresh 5/5 review of `9d9737b496911a6054d0ea22d1b42b28b115e9ab` with no open findings. - CI on `938ad1665` exposed an integration regression with the newly merged durable startup saves: the personal preference gate blocked the task page before its first session lookup. The guarded title editor now mounts while that lookup is pending, with presentation-specific tools still gated. All 216 affected component tests pass. The unchanged five-test `disconnect-write-recovery.spec.ts` browser suite passes against a newly launched disposable local-trusted PostgreSQL instance on port 33219, including the exact failed startup-save journey. No live provider, hosted authentication or production proof is claimed. The helper-owned test server exited and the port has no listener. UI typecheck, built-browser assets, token and whitespace checks pass. - Initial durable-write merge tests found duplicate imports and unscoped draft fixtures, fixed before 416 focused tests passed. A sandbox typecheck could not create its IPC socket; the supported rerun passed. The first revision-link test used a wrong Return to latest button assertion, corrected before the 113-test pass. The startup E2E reproduction initially still failed because an outer shell gate also blocked the page; the subsequent fix passed all five cases without changing the test or its timeouts. A focused loading-header assertion initially used an incorrect fixture title; correcting it preceded the 216-test pass. These failures remain recorded. - Earlier routing gates fell back to GitHub-hosted runners while master was moving or conflicted. Superseded queued runs were cancelled, then the `938ad1665` routing gate explicitly selected the ephemeral RunsOn fleet. Its CI actively executed and completed most tests; only the diagnosed startup E2E shard and aggregate failed, alongside the valid Greptile finding. Auxiliary GitHub-hosted workflows also completed successfully on earlier heads, so queueing alone is not an administrative blocker. Fresh [CI on `9d9737b49`](https://github.com/paperclipai/paperclip/actions/runs/40744012678) is the acceptance target; the previously failing startup E2E shard now passes. A different signoff E2E shard failed during executor resubmission when its fixture run was cancelled as `issue_continuation_waiting_on_review`. The exact unchanged journey passed locally (1 test, 46 seconds), and the preceding head with identical server/signoff code passed that CI shard. This supports a bounded rerun but does not establish the timing failure's root cause. After the workflow finished, GitHub accepted one rerun of that failed shard and its dependent aggregate; both passed, completing all checks on that head. The failure log is retained at `/tmp/pr17983-9d9737-e2e7.log` and the local reproduction at `/tmp/pr17983-signoff-repro.log`. No assertion or timeout was weakened. - On the first merge, full workspace typecheck/build, all 9,622 UI tests, 267 tests across seven affected server suites, two migration checks, map validation, token gates, and the PR diff whitespace check passed. Greptile then found duplicate local task-search filtering and lost completed item-review detail. Both are fixed, with 42 passing focused UI tests and passing UI typecheck/build. - A subsequent Greptile review found attachment confirmations without working links and mobile history without a visible host. Confirmation layouts now derive the authenticated immutable-content endpoint from the attachment ID, and Needs You uses the existing full-screen mobile sheet. All 119 focused tests pass, including missing/stale display URLs, both review layouts, and both mobile history entry buttons. UI typecheck/build and token checks pass. Full workspace typecheck/build also pass after the second master merge. - Greptile gave head |
||
|
DottaandPaperclip
|
6b9e5f3a80 |
fix(evals): enforce qualified ACPX profile admission before paid turns (#17964)
## Thinking Path
> - Paperclip coordinates agent work through Runner.
> - Controlled evals route model calls through a budget gateway.
> - The definitions pin each ACPX runtime identity.
> - A stale qualified profile currently reaches a paid turn before
scoring rejects it.
> - The Runner CLI already checks an exact expected profile before it
creates a provider.
> - This change forwards that existing admission argument through the
controller.
> - Profile drift can then stop before model spending.
## Linked Issues or Issue Description
Refs #17943 and #17929.
**What happened?**
The worker in run 40618022700 passed all four behavior checks and
retained three final request receipts. Its grader then rejected the old
Claude SDK pin. The controlled wrapper could not pass the existing
`--expected-acpx-profile` argument to the built CLI.
**What did you expect to happen?**
The built CLI must reject a stale expected profile before provider
creation.
**Steps to reproduce**
Run a controlled ACPX cell with a definition that pins a different SDK
from the built Runner. The old wrapper accepts only request and output
arguments. It cannot enforce the profile admission supplied by updated
definitions.
**Paperclip version**
The observed run used
|
||
|
DottaandPaperclip
|
e4b30163bb |
fix(evals): preserve the complete trusted controller artifact inventory (#17943)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/103/changes/paperclip/103 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages real agent work. > - Controlled Runner evaluations record model spend and traces. > - A trusted build creates one portable controller with a complete file inventory. > - GitHub artifact upload omitted one hidden dependency metadata file. > - The next job correctly refused the incomplete inventory before any paid worker started. > - This change preserves all inventoried bytes without relaxing verification. ## Linked Issues or Issue Description Refs #17889, #17929. **What happened?** [Run 40585138437](https://github.com/paperclipai/paperclip/actions/runs/40585138437) failed at copied controller verification. Its artifact contains 294 files; its signed inventory expects 295. The only missing file is `server/node_modules/protobufjs/node_modules/@protobufjs/pool/.npmignore`. No extra files exist. Every paid worker was skipped and no model request ran. **Expected behavior** The trusted controller artifact preserves its full inventory, including hidden package metadata. The strict verifier rejects any actual change or missing file. **Steps to reproduce** Build the portable runtime, upload the directory with the action's default hidden-file exclusion, then verify the downloaded artifact against the original receipt. ## What Changed - Enable hidden files only on the fresh trusted controller code upload. - Download that exact upload and run a real artifact-service round-trip test before any private preparation. The test fails if the upload drops hidden files again; it tests the actual transport instead of matching a YAML flag. - Keep the strict byte inventory and trusted source/receipt checks. - Document why this code artifact differs from private attempt evidence, which still uses the protected archive. ## Verification - Exact retained artifact comparison: one missing `.npmignore`, zero extra files. - `actionlint .github/workflows/runner-protocol-live-evals.yml` passed. - Protocol workflow security tests: 14 passed. - The artifact-service integration test is explicit-only locally. The controlled workflow supplies the real downloaded artifact, trusted source and receipt digest. - No product runtime, provider routing or grading code changed. CI runs broader checks. Fresh controlled workflow qualification follows merge. ## Risks Hidden files are allowed only in the code-only output of the provider-credential-free build. Raw private evidence and credentials do not enter that upload. The artifact verifier still requires every file and SHA-256 from the trusted receipt. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0a08d37aa0 |
feat(runner): recover controller authority without interrupting execution (#17097)
## Thinking Path > - Paperclip manages agents and owns their business authority. > - Runner owns execution transport, acknowledgements and replay. > - A replacement controller must recover that transport without starting another provider turn. > - An external store can retain the required control records without copying streamed text. > - A duplicate local journal adds synchronous disk work without providing the replacement server's authority. > - This PR adds the optional recovery contract and its standalone proof. Server adoption and browser delivery have separate owners. ## Linked Issues or Issue Description **What existing behavior does this improve?** A surviving runner can reconnect to a replacement controller with its admitted process, session and active turn. **Current behavior** Local controller files do not provide recovery on empty replacement storage. Rewriting those files after an external checkpoint is installed also blocks normal controller work. **Proposed behavior** An optional callback stores opaque checkpoints. The replacement restores that checkpoint before it constructs the controller. Installing the callback makes the external store authoritative. Local storage holds a marker that requires restoration. Consumers without the callback retain their file journal. PR 1 of four: #17097 → #17439 → #17455 → #17456. Related work: #17390 and #17361. The pre-split reference remains `codex/hot-upgrades-reference-2026-10-24`. ## What Changed - Add opaque checkpoint encoding, restoration, incremental persistence and authenticated sparse recovery cursors. - Keep authentication, commands, semantic inputs and lifecycle events behind their commit barriers. Share semantic-call ownership between live dispatch and recovery. - Keep streamed text out of recovery checkpoints. Preserve bounded retention, original deadlines and admitted process identity. - Stop normal local journal writes after external persistence is installed. Fail closed on a local marker until external restoration occurs. Preserve current master's staged file writes for standalone consumers. - Keep provider-native admission and authorization behind adapters. Preserve historical Copilot attestations and bind the merged launch and transport sources in v68; preserve published v66 and prior development identities. - Add the one-process scripted executor fixture and deterministic outage, replay, failure and backpressure tests. Add no production test endpoint or auxiliary process. ## Verification The completion-recovery correction passes three restart regressions that fail before its respective transport/driver fixes. All 230 controller/attestation cases, all 302 Codex transport cases in the full repeat, and all three one-process fixture cases pass. The broader Codex/Dot/Muse run passes 326 cases and times out one existing mid-goal case; its isolated retry and complete Codex repeat pass on unchanged source and binaries. The initial timeout remains recorded. Conformance and replay parity pass. Historical v65/v66 attestation bytes remain frozen; v67 binds the changed transport. The local runner-only API-authority lane passes 2,027 cases but its 47 HTTP cases cannot initialize embedded Postgres on this Mac. The integrated owned-database HTTP path is verified in PR 2. At preceding head `0d8d1b8250`, the Actions workflow passes after one retry of runner-service shutdowns. Greptile is 5/5 with no unresolved threads. The interrupted attempt is retained. All checks on that preceding head are terminal and green/neutral/skipped. Run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/control-plane/durable-prp-control-plane.test.ts src/live/runnerd-codex-transport.test.ts --maxWorkers=1` and `node packages/paperclip-runner/scripts/copilot-profile-identity.mjs --check`. The [qualification record](https://github.com/paperclipai/paperclip/blob/codex/hot-upgrades-qualification/doc/plans/2026-10-22-hot-upgrade-qualification.md) retains exact revisions, failed attempts and qualification limits. The local cost campaign observes zero hot-path local journal fsync calls. It does not qualify a native harness or hosted deployment. Master reconciliation `d5fc443626` preserves the published v66 and prior development identity bytes and binds the combined transport in v68. All 290 focused runner/profile cases and runner typecheck pass. The first sandboxed invocation cannot open loopback listeners; the permitted repeat passes. Fresh CI and Greptile are pending for this updated head. Current merge candidate `4fb3d1835b` also fixes public-reader publication before external commit. New held/rejected-write cases fail before correction and pass afterward. The full controller repeat passes 168 cases. The combined lane passes 390 other recovery/transport/profile/fixture cases; its first two overlap fixture failures are retained and corrected to observe callback entry instead of provisional state. Runner TypeScript build and typecheck pass. Current-head GitHub Actions is green and Greptile is 5/5 with all threads resolved. The superseded master-reconciliation CI shutdown remains recorded. ## Risks The persistence callback is an acknowledgement barrier. Failure must stop delivery. A consumer that installs external persistence must restore its committed checkpoint before restarting; the marker is not a recovery snapshot. Sparse display recovery does not reconstruct acknowledged text or repeat execution. This slice has no Postgres schema, application authorization or browser behavior. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
35820cf398 |
fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929)
## Thinking Path > - Paperclip must account for paid eval work. > - The maintained Runner workflow distributes one immutable controller manifest. > - Its cost gate currently requires vendor proofs for every operating charge. > - Model-spend qualification must also work under an explicit policy that keeps operating charges separate. > - This PR applies the shared model-readiness policy to the hosted controller. > - It retains every operating hold and still rejects missing model receipts and incomplete workers. ## Linked Issues or Issue Description Refs #17928, #17889. **What happened?** Runner qualification requires full-lifecycle vendor operating proofs even when the operator selects model-request readiness. That blocks the model-token qualification path. **Expected behavior** An explicit `overhead.accountingReadiness: "model_requests"` policy permits absent operating cost proofs. Supplied proofs still validate. Complete final model receipts are required. **Steps to reproduce** Prepare a two-worker hosted manifest with the explicit policy and no vendor cost proofs. Preserve missing-worker allocations, then complete both workers and inspect the private archive summary. A second test returns a provider refusal without terminal usage and requires failure with the full hold. ## What Changed - Apply the shared explicit policy to hosted capacity and observability cost-proof admission. - Preserve pricing windows, model bounds, immutable source/request/runtime identities, trace quotas and every allocation. - Reject zero or unresolved model requests in this policy, even if a request has a provisional bound. - Retain operating amounts and pending reconciliation through campaign and private archive summaries. - Document the separate model-readiness contract. Preserve strict behavior when the field is absent. ## Verification - Regression first failed at the original vendor-bound gate. - All 80 controlled-eval Node checks pass, including source and copied portable runtime checks, archived partial/missing workers, credential exclusion and model receipt failure. - Product readiness prerequisite #17928 has its own eval unit/typecheck and full CI checks. - No paid workflow, credential change or recurring activation is part of this source PR. ## Risks This opt-in policy does not prove a full-lifecycle operating-cost ceiling. The report retains operating holds and pending reconciliation. Default policies retain the existing vendor-proof gate. Source and worker identity checks remain required. A model refusal without final usage now fails readiness even if its reservation has a reviewed bound. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8548173fc9 |
feat(evals): connect hosted Runner costs and traces (#17889)
## Thinking Path > - Paperclip manages AI agents and measures their behavior with Runner evaluations. > - The cost controller, native tracing, private receipts and portable runtime are on master. > - The maintained live workflow still launches the provider CLI directly. > - Independent workers need one shared selection and disjoint budget allocations. > - Reports must keep unresolved allocations when a worker produces no evidence. > - This PR connects the maintained workflow to the existing controller and private archive. > - Operators can inspect known costs, conservative holds and trace evidence before qualifying paid recurrence. ## Linked Issues or Issue Description Refs #17452, #17469, #17493, #17506, #17523. **Current behavior** The live Runner workflow does not use the merged cost/tracing controller. Attempt-only aggregation cannot reconstruct a campaign when every worker archive is absent. Ordinary Actions artifacts in this public repository are readable by authenticated public GitHub users. **Proposed behavior** Controlled runs prepare one manifest, bind every selected request, enforce disjoint allocations and retain evidence in private storage. The report starts from the full retained manifest. Missing workers keep their allocations. The workflow rejects retries, unsupported providers, changed requests and absent or stale operating bounds before provider work. **Reason and benefit** The maintained workflow needs auditable costs and traces, including failures and missing receipts, before paid recurrence can be qualified. ## What Changed - Add controlled mode as the manual default and the only scheduled mode. Legacy mode remains explicit for other manual provider qualifications. - Prepare the full request selection and one shared manifest. Bind current run, first attempt, code, definitions, executable hashes and exact profile allocations. - Verify the copied controller and manifest before provider credentials. Require the controlled target to equal the trusted workflow revision before private archive access. - Reuse private archive retention for the controller policy, cell attempts and canonical report. AWS credentials remain transfer-step outputs. Controlled raw evidence is never uploaded as an Actions artifact. - Replay a retained manifest with partial or empty worker evidence. Count operating holds once and preserve known partial charges. - Pin the controller to an explicitly reviewed compatible lockfile digest. Reject resolver or byte drift before a frozen, script-free install into a job-local store. Record the lock digest in the runtime receipt and verify it again in private preparation. - Build the portable controller without secrets, private environment access or OIDC. Private preparation verifies that artifact and performs no dependency install. OIDC jobs use no mutable dependency resolution or shared cache. - Add synthetic source/copied-runtime tests, executable workflow admission tests, private archive tests and an operator runbook. ## Verification - Controller, campaign, portable runtime and workflow tests: 104 passed. This includes real aggregation from an isolated reporter without source dependencies, safe diagnostics, driver drift, distinct evidence binding and reviewed-lock drift. Real private Codex/Claude requests also pass through an independently verified copied runtime with fake transports. - Private transport and workflow contracts: 35 passed. Executable/static workflow guards: 13 passed. - Paired private report tests: 166 passed; Evalbook discovery: 7 passed. - `pnpm -r typecheck` and `pnpm build`: passed. - `pnpm test:e2e:runner:typecheck`, `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check`: passed. - `pnpm test:run` hit repeated local embedded PostgreSQL initialization failures in unrelated DB-dependent server tests. Its owned process tree was stopped after preserving the log. This is a local verification limitation; current-head CI remains required. - An earlier head hit one unchanged Rust cleanup assertion. The exact test passed locally and the final head passed the full Rust lane; the initial failure log is retained. - On the final head, one unchanged chat idle-queue assertion failed once in CI. Its single scoped retry passed. All test jobs and the final aggregate gate passed on commit `2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt 2). - No paid model, Braintrust or cloud calls. No live variables, secrets, schedules or resources changed. ## Risks - Controlled mode fails closed until a reviewed `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private eval SHA exist. `all` does not silently become a supported subset. - The reviewed policy must cover setup, model requests, trace/gateway processing, archive transfer and retained storage. Hash binding does not establish vendor billing bounds. This PR does not enable recurrence or release historical holds. - The controlled policy must pin `controllerLockSha256`; it never accepts a resolver-generated digest automatically. No lockfile is committed. A changed compatible lock needs review. - Legacy browser verification now requires the committed frozen lockfile. A stale lockfile fails closed; controlled preparation has no dependency install. - Workflow cancellation can prevent the final report. The full manifest is retained before provider work and remains the starting point for reconciliation. - The workflow exceeds 1,000 lines because the shared shard anchor, dependencies, environment expressions and credential boundaries belong to one declarative job graph. Request preparation, accounting, archive safety and runtime verification stay in separately tested modules. ## Model Used OpenAI Codex, GPT-6-based coding agent with tool use and code execution. The host does not expose an exact deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the relevant tests locally and they pass; the broad local suite limitation is documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the final head - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups; the secondary reviewer is also 5/5 - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
27a87c0f22 |
perf(runner): keep routine journal flushes off the server thread (#17813)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runners persist incoming events in the server process. > - Routine progress and usage events can cause many full journal writes. > - Synchronous file flushes stop that process from serving unrelated requests. > - Output deltas already use asynchronous writes, but other frequent events still block. > - This pull request stages routine events privately and publishes them after persistence. > - Critical lifecycle events keep their existing ordering. ## Linked Issues or Issue Description Refs: #17745, #17514. **What happened?** Page navigation, task saves, and archive requests stalled during native runner output. A post-fix CPU profile still spent about 7 seconds of a 19-second window in synchronous event journal flushes. Progress and usage events caused much of this traffic. **Expected behavior** Routine event file writes must let unrelated HTTP requests run. The durable cursor must advance only after persistence. Completion and semantic tool admission must retain their ordering. **Steps to reproduce** Run a native task that emits frequent tool progress and usage events. Send HTTP requests while its volume flushes are slow. The controlled tests hold a real file flush and send a real HTTP request during that hold. **Paperclip version or commit** Reproduced after #17745 on master. ## What Changed - Stage priority 1 and 2 event snapshots with asynchronous file writes and flushes. - Keep candidate events out of the public in-memory journal until the durable commit finishes. - Include a pending event in a command's existing durable write. Command bursts do not cause repeated event writes or business callbacks. - Reject stale candidates after an authenticated connection replaces their owner. - Keep priority 0 events, all run and turn lifecycle events, and semantic tool input persistence synchronous, including terminals labeled priority 1. - Extend slow-volume tests for progress, usage, critical events, concurrent commands, reconnects, write failures, drain, and replay. - Update the durable event delivery documentation. ## Verification - Controlled regression: the previous implementation failed the new progress and usage slow-flush cases and exposed pending delta state. The updated implementation passes all sixteen controlled cases. - All 137 durable control-plane tests and three saved-continuation cases pass. The continuation cases use the actual Rust bridge. - Review regressions failed before the correction: a priority-1 terminal took the asynchronous path and command writes did not absorb the pending event. The correction passes these tests, a two-command burst with one staged write, and a directory-flush failure after command/event rename. - An authority-boundary regression also failed before its correction. Explicit snapshot transactions now retire pending coalescing, so a later command cannot publish an older event; the test verifies replay after restart. - An initial version made lifecycle events asynchronous too. The saved-continuation tests caught missing final run receipts at completion. Keeping priority 0 events synchronous fixes those failures. This version does not claim asynchronous lifecycle support. - Repository typecheck and build pass locally after the review fixes. At the previous head `886f2b961f035071bd947b5116b1f7a2727ec17f`, run `40404783081` passed the full CI suite, including 4,024 package runner tests and 96 native-runtime tests. Final-head CI run `40409537748`, attempt 2, passed. All 46 substantive jobs and both aggregate gates succeeded. Runner lanes passed 4,032 package tests and 96 native-runtime tests. The retry carried forward the successful substantive jobs from attempt 1 and completed both aggregate gates. - Greptile reviewed final head `735245931450810246e3008231a29fd56d615f67` at 5/5 with no actionable findings. The separate Paperclip review execution ended neutrally without a new assessment. Its earlier command-churn finding is fixed and its outdated thread is resolved; the documentation and priority-1 coverage comments are also addressed. - The local wider runner suite passed 449 of 450 tests. The retained-terminal replay case fails on both this candidate and unchanged master at the same maintenance-proof check. Both local runs leave an old terminal receipt pending behind the event backlog. These failed runs remain recorded; they are not counted as passes. - The broad local test command also failed one unchanged workspace-runtime port-allocation case. That case passed in isolation. Its cause is unresolved. The broad local run was stopped after the complete CI test groups passed; no full local-suite pass is claimed. - Live staging acceptance remains pending. - The tests hold a real `FileHandle.sync`, verify an actual HTTP response during the hold, and check disk and memory cursors. They also verify command preservation and idempotent replay. ## Risks - Asynchronous writes add time for concurrent commands or reconnects to arrive. Commands absorb the event in their existing write. Revision checks prevent stale publication. Other authority changes leave the unacknowledged event for replay instead of retrying full snapshot writes. - ACKs still wait for the file flush, atomic rename, and directory flush. Write errors still fence further writes until reload. - Critical commits, JSON serialization, rename, and directory flushes still run synchronously. A command's coalesced event uses that command's existing synchronous commit. This change does not eliminate every possible source of latency. - No schema, API, dependency, or wire contract changes are required. ## Model Used OpenAI GPT-6 in Codex, with reasoning, tool use, and code execution. The session does not expose a more specific deployment model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
507dc70be2 |
fix(evals): bind hosted controller admission to the first attempt (#17523)
Bind hosted controller admission to the repository, GitHub run, first attempt, workflow revision and selected execution before credential reads or provider launch. Keep offline evidence replay independent of the current worker environment. Preserve the single-project boundary editor and refresh project-access data with agent-detail updates.
Verified head:
|
||
|
DottaandPaperclip
|
8c702d59db |
feat(evals): package a portable Runner controller runtime (#17506)
Package a verified portable Runner eval controller with complete bounded file receipts, copied dependency closure, private evidence replay and symlink-safe CLI entrypoints. Preserve fail-closed cost holds, native trace receipts and all parent security repairs.
Verified head:
|