mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
ee72132a44e2da2cb9539b40f88d057ea7a3cdd6
2152
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin FoleyandClaude Opus 5.5
|
ee72132a44 |
fix(daytona): bound one-shot exec requests by their command deadline (#19298)
Bound the Daytona one-shot executeCommand request at its command deadline plus 5 s and return an attributed timed-out result, instead of waiting for the host RPC deadline. Bound the stdin cleanup the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
7f5d0793bd |
fix(daytona): record a bounded errno when a sandbox command fails (#19297)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Daytona sandbox provider copies the agent's workspace back to
the host after each turn (syncOut)
> - When the sandbox `tar` fails in the `archive_create` step, the
restore diagnostic records only `exitCode: 2` and `errorCode: "unknown"`
> - The command output is not sent across the plugin RPC boundary, so an
operator cannot tell a full disk from a permission error or a path that
is too long
> - This pull request maps the system error text at the end of a failed
sandbox command's output line to the existing allowlisted `errorCode`
> - The benefit is that the next failure records the real errno, and the
output text stays private
## Linked Issues or Issue Description
Refs #18172
Refs #18212
**What happened?**
A hosted Daytona run failed its workspace restore. The `archive_create`
step exited 2 on all three attempts. The run log recorded
`{"step":"workspace_transfer","transferStep":"archive_create","transferFailureKind":"command_failed","exitCode":2,"errorCode":"unknown"}`.
Two causes are possible. The sandbox disk was full, or a path was longer
than PATH_MAX, for example from deeply nested GitHub helper config
directories before #18212. The diagnostic cannot tell these causes
apart.
**Expected behavior**
A failed sandbox command records a bounded errno such as `ENOSPC`,
`EACCES` or `ENAMETOOLONG` when its output shows one. The output text
itself must not be recorded.
**Steps to reproduce**
1. Run `performSyncOut` against a sandbox whose `tar -c` exits 2 with
`tar: ./x: Cannot open: Permission denied`.
2. Read the environment sync diagnostic from the thrown error.
3. See that `errorCode` is `unknown`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
55389554c2 |
fix(runner): paginate workspace discovery before journal persistence (#19311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents discover reusable workspaces through `list_workspaces`. > - The tool returned every authorized workspace in one result. > - A large inventory exceeded the durable command journal limit. The result stayed pending, and session cleanup blocked the next turn. > - This pull request limits discovery in SQL and returns explicit continuation metadata. > - Agents can inspect every authorized workspace without sending the full inventory in one command. ## Linked Issues or Issue Description Related: #15749 introduced the task workspace tools. No duplicate pagination fix was found. **What happened?** A native workspace discovery call returned a multi-megabyte inventory. Persisting its result failed with `command_payload_too_large`. The preceding answer could still be saved, but cleanup retained the unsettled tool call. A later turn failed before provider work with `native_session_cleanup_quarantined`. **Expected behavior** Workspace discovery must return a manageable page. Agents must be able to request the remaining authorized rows. Existing visibility checks and journal integrity checks must stay effective. **Steps to reproduce** 1. Create 2,000 task-selectable workspaces with ordinary repository paths. 2. Call `list_workspaces` from an admitted native run. 3. On the old code, the real API regression returns 869,781 bytes. The command journal allows about 510 KiB. 4. On the repaired code, follow `nextCursor` to inspect all rows. Pages contain at most 100 rows and fit within 128 KiB. **Paperclip version or commit** Reproduced on source commit `7351a9d74f` before the repair. **Deployment mode** Isolated source checkout with real PostgreSQL and authenticated native agent APIs. Hosted deployment and incident recovery remain separate. ## What Changed - Validate workspace list limits and UUID cursors with a JSON Schema pattern supported by strict validators. Return `workspaces`, `limit`, `returnedCount`, `hasMore`, and `nextCursor`. - Apply company, visibility, lifecycle, and cursor conditions before the SQL limit. Order pages by workspace ID. Fetch one extra row to detect continuation without a count query. - Fit encoded pages within 128 KiB, including JSON escaping and Unicode. If one summary cannot fit, return explicit string previews with `truncatedFields`. The authorized detail API retains its full record. - Keep existing unpaged summary callers on their array response and recent-use order. - Update the native tool descriptor, generated contracts, catalog fixture, and execution documentation. - Add regression coverage for large inventories, complete pagination, stable pages, private and foreign rows, invalid inputs, and legacy summary behavior. Add an opt-in two-turn live Codex acceptance check. ## Verification - Red before repair: `task-workspace-api.test.ts -t "pages a large authorized workspace inventory"` failed because the result was 869,781 bytes instead of below 128 KiB. The transport regression also failed at 2,284,671 bytes. - Green after repair and review changes: 120 tests passed across the workspace API, transport, routes, native authority, coordinator, historical-question, and Codex vertical-slice suites. They cover complete discovery, byte-sized pages, oversized individual summaries, full detail retrieval, private/foreign/lifecycle exclusions, and strict schema compatibility. - Journal safety: two focused durable control-plane tests passed. Oversized result rejection and lossless valid result delivery remain effective. - Tool catalog and reconciliation: 30 tests passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed again on final commit `bc10880da6`. Server typecheck also passed after the review changes. The original affected suites passed on the rebased first commit (31 tests), and the expanded review regression set passed afterward (120 tests). - Real provider acceptance passed again on final commit `bc10880da6`: local macOS arm64, rebuilt native Runner, `gpt-6.1-sol`, high effort, per-turn lifecycle, and 2,000 persisted workspaces. Both turns succeeded. Persisted call inputs and results prove that the second page uses the first result’s cursor, both calls succeed, their IDs do not overlap, and the encoded results stay within 128 KiB. The second turn kept the provider conversation and file. Both task locks were clear after completion, and no recovered execution failure was recorded. Duration: 166.92 seconds. - Full local `pnpm test:run` stopped in the chat shard: 330 tests passed and one Slack registration test failed with `socket hang up`. That exact test passed in isolation without changes, and the final-head CI chat shards passed. A broader local managed-Slack follow-up produced no result after about eight minutes and was interrupted; it is not counted as a pass. The original full-suite failure remains recorded. - The first CI head failed native tool-schema compilation because strict validators did not recognize `format: uuid`. The revised descriptor uses a UUID pattern. The previously failing coordinator, native vertical-slice, authority, and history suites now pass locally. The stronger live cursor-receipt acceptance check passed. Final-head CI completed on `bc10880da6`: 57 checks passed and five optional checks were skipped, with no failures or pending checks. Greptile reviewed that commit at 5/5; both prior findings are resolved. - Dependency cost: paged discovery runs the existing visibility-scoped summary query with an ID cursor and at most 101 returned rows. It adds no count query, filesystem scan, provider call, or background worker. ## Risks - Native `list_workspaces` now returns a page object instead of an array. Individual summaries can be explicit previews when a single record exceeds the byte budget. Its advertised descriptor and generated contracts change together. Agents must follow `nextCursor` until null for a complete inventory. - Each page rechecks current visibility. Concurrent insertions, deletions, or access changes require a fresh scan when a complete point-in-time inventory is needed. - No migration or journal-limit change. An existing cleanup quarantine still requires normal recovery after deployment; this PR does not bypass it. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, code editing, and test execution. The exact served model revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
a6f786479e |
Runner: verify provider eligibility before parking (#19133)
## Thinking Path > - Paperclip coordinates AI agents and their execution. > - Hosted Runners can retain a provider process between turns. > - The host must prove that process is idle before allowing its VM to sleep. > - A completed turn alone does not rule out an active goal or trailing provider work. > - This PR adds a conservative park-readiness hook for the existing Codex process. > - The later host-control consumer can use this proof with its identity and sequence fences. ## Linked Issues or Issue Description **Current behavior** There is no provider eligibility hook for host parking. The earlier implementation in #19060 also missed an active goal that arrived during its tail drain. **Proposed behavior** Default every executor to ineligible. Permit the surviving native Codex process only when its completed-turn authority still matches and it has no live or unsettled work. **Reason and benefit** Keep active goals and unresolved provider obligations awake. Review the provider proof separately from socket handling and the native park/reclaim loop. ## What Changed - Add `host_park_ready` with a default false result. - Refresh the configured credential observer before checking pending delivery. Refuse parking during credential delivery, without selecting or restoring a provider. - Verify the current Codex process and completed-turn identity; reject startup, active turns/goals/tools, and unsettled event queues. - Require completion actually observed by that process. Accepted-finish reconciliation still preserves ordinary result authority without qualifying a restored process to park. - Refuse known helper lineage, whose diagnostics do not prove that helper work has finished. - Drain bounded trailing output through the existing accounting path. Retain goal updates and clears in durable state and normal event delivery, then recheck live goal state. - Add process tests for normal eligibility and the late-goal race. Assert that a restored process is ineligible until it completes an authoritative turn in its new generation. This PR is based on master, which contains the merged Codex (#19132) and ACPX (#19153) recovery foundations. The private socket codec in #19129 is merged; #19140 consumes this proof in the shared native loop. This PR does not advertise parking support. A separate ACPX readiness change must prove its live sidecar/session has no active or unsettled work. Both Codex App Server and qualified ACPX are required in the remaining park/reclaim, Core restart, eviction, and live sleep/wake acceptance phases. ## Verification - `cargo test --offline --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test codex_provider hosted_park` — 5 passed. - `ordinary_recovery_restores_a_completed_idle_session` passes and asserts that the genuinely restored provider cannot park before it completes an authoritative turn in its new generation. The fixture proves two provider starts and only one admitted turn. - The initial live-goal recheck mutation failed as expected. Review then identified that goal changes also needed durable retention. The new activation and clear cases verify saved goal state, retained events, and eligibility after delivery. - Existing goal, active-goal attachment, warm-tail, and late-accounting process regressions passed after sharing the goal event path. - The initial late-goal fixture timed out because it omitted `--durable-turn-ids`, which enables its existing first-turn gate. Added that fixture switch; both park tests then passed. - The credential-observation regression fails before the fix and passes afterward. Three focused credential cases prove durable replay and acknowledgement, fresh ACPX startup, and readiness without provider selection. The three provider parking checks also pass after the fix. - Full regression testing is left to GitHub CI. - Changed lines scanned for secrets, personal data, and private deployment references before push. Current head `28d41b083f58ee93db4305b1217719796bf3171d` adds one scoped review-fix commit to readiness `e8ba82374e0f72c82ac3195d4b1b3798c9072bab`, based on master `3d0d178ff7051aa44e07cd873fd4281357e0d815` without duplicating either recovery foundation. The PR remains eight parking-specific files. Review identified quiet helper work and reconstructed accepted-finish authority that could incorrectly qualify. The provider now records its actually observed terminal authority separately in memory, and parking refuses known helper lineage in both the saved state and the live inventory after draining. No durable field, lifecycle, public API, or ordinary recovery policy changed. Both process regressions fail when the parking guards are removed and pass when restored. Seventeen focused checks pass, including ordinary recovery, helper requests, receipt replay, credentials, and ACPX ineligibility. Rustfmt, diff and added-line scans passed. All exact-head CI is green: [PR run](https://github.com/paperclipai/paperclip/actions/runs/42087773928), [Runner Windows](https://github.com/paperclipai/paperclip/actions/runs/42087773720), and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42087773712) passed; Storybook was skipped. Standard review 15 scored 5/5 before the authorized follow-up [Apex review](https://github.com/paperclipai/paperclip/runs/127471966630). Apex review 16 also scored [5/5 on this exact head](https://github.com/paperclipai/paperclip/pull/19133#issuecomment-6645371152), with zero new comments and zero unresolved threads. On this current head, the first server shard 11/12 [job 127458190155](https://github.com/paperclipai/paperclip/actions/runs/42087773928/job/127458190155) failed the unchanged decision-model concurrency test: after releasing the database lock, its `waitFor` observed one provider call instead of two. The identical file passed all 37 tests on the prior reviewed head. One targeted unchanged-head retry [job 127466020677](https://github.com/paperclipai/paperclip/actions/runs/42087773928/job/127466020677) passed all 1,779 tests, including those 37 decision-model cases; 13 tests were skipped. No code, test timeout, or CI configuration changed between attempts. The original failure's cause remains unestablished. On prior reviewed head `e8ba82374e0f72c82ac3195d4b1b3798c9072bab`, the first `General tests (workspaces-c)` job [127389771745](https://github.com/paperclipai/paperclip/actions/runs/42068371976/job/127389771745) failed the unchanged database SIGTERM test at `packages/db/src/test-postgres-command.test.ts:44`: `JSON.parse` read empty readiness-file content immediately after an existence check. The fixture writes directly to that path, leaving a create-before-content race. One targeted unchanged-head retry [127398676711](https://github.com/paperclipai/paperclip/actions/runs/42068371976/job/127398676711) passed all **313 tests in 72 files**, including SIGTERM and SIGINT. No source or timeout changed between attempts. On the superseded `ca088abd552b` head, one Cloud sign-in recovery case returned to the expected task URL but missed its heading within the five-second assertion window; the other 40 cases passed. One targeted unchanged-head retry (job 127136265280) passed all **41 cases**. No test timeout or application code changed between attempts, and the first failure's cause remains unestablished. The superseded c7 parent failed eight retained-cleanup settlement cases because prepared restoration started a provider during a drain-only epoch. The corrected parent preserves ordinary prepared behavior; the new completed-source path excludes prepared providers. The parent reports all 19 settlement rows passing. This PR adds no prepared-session proof or restoration policy. ## Risks Known helpers remain conservatively ineligible even after their progress diagnostics: this slice has no authoritative helper-completion ledger. The eligibility check drains the existing bounded provider tail and can persist late accounting. That accounting must settle before parking succeeds. This hook alone does not prove VM suspension or vendor goal behavior across a process restart. Live sleep/wake qualification remains a later rollout check. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), High reasoning, through Codex with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
377e98f1a2 |
test(runner): keep the production close budget for recovered prepared-descriptor closes (#19309)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner closes a native session only after it proves
that the provider drained and the runner suspended
> - The test "runs an environment-prepared descriptor without Core
filesystem preparation (rotated-recovery)" closes recovered transports
with a 5s close grace
> - A 5s grace leaves only 2.5s for the provider drain, and a recovered
successor needs several drain round trips
> - On a loaded CI runner with the debug runnerd, the drain does not
finish in 2.5s, so close correctly fails closed and the test fails
> - This pull request makes those closes use the production close budget
> - The benefit is that the Runner Vitest lane stops failing on
unrelated PRs, and the close assertions stay the same
## Linked Issues or Issue Description
No issue exists. The problem, in the bug report format:
**What happened?**
The Runner Vitest lane failed on PR #19265 (run 42075789869, job "Verify
Paperclip Runner (vitest package)"). The change in that PR is not
related to the Runner:
```
FAIL src/live/runnerd-codex-transport.test.ts > runs an environment-prepared descriptor without Core filesystem preparation (rotated-recovery)
NativeSessionCloseUnrecoverableError: provider_transport_failed: runner did not durably suspend before checkpoint
❯ src/live/runnerd-codex-transport.test.ts:2768:21
Serialized Error: { settlement: { runnerSuspended: true, providerDrained: false, suspensionState: null, ... finalProviderState: null } }
```
**Expected behavior**
The test fails only when recovered close behavior is wrong. It does not
fail because the CI machine is slow.
**Steps to reproduce**
1. In the rotated loop of the test, set the close grace to 700ms instead
of 5_000.
2. Run `npx vitest run src/live/runnerd-codex-transport.test.ts -t
rotated-recovery`.
3. The second (recovered) close fails with the same error and the same
settlement (`runnerSuspended: true`, `providerDrained: false`,
`suspensionState: null`).
Root cause: `runnerCloseDeadlines` keeps half of the grace (max 2.5s)
for suspension. The rest is the preparation budget for the provider
drain. With `closeGraceMs: 5_000`, the drain gets 2.5s. In a local
measurement, the recovered close needed 4 `runner.drain` commands and
about 440ms on an idle machine. Each drain is a full runner round trip,
because a drain receipt is `true` only after the controller ACKs the
prefix that the previous drain moved. On a loaded CI runner with the
debug runnerd binary, those round trips use more than 2.5s. The
transport then fails closed, which is correct product behavior.
Production uses a 10s default grace (7.5s for preparation, with the
drain capped at 5s).
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
03eb49053f |
test(runner): bound Codex provider event waits by a hang ceiling, not 5s (#19306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust runner drives provider processes such as Codex, and CI tests it on Linux, macOS and Windows > - The "Runner Rust (Windows x64)" job fails at random on pull requests that do not touch the runner > - The failures come from Codex provider integration tests that wait for fake-provider events under a fixed five-second deadline > - On a loaded Windows runner, process spawn, antivirus scans and fsyncs of durable state make these exchanges take longer than five seconds, so the wait stops before the event arrives > - This pull request keeps each wait on its condition and changes the deadline to a shared hang ceiling > - The benefit is that unrelated pull requests do not fail on this job, and a real hang still fails the test ## Linked Issues or Issue Description **What happened?** The "Runner Rust (Windows x64)" job failed on pull requests that do not change the runner. Examples: - Run 42073331106 (PR #19273) and run 42058757139: `rejected_declared_completion_stays_failed_when_the_provider_completes` panicked with `called Option::unwrap() on a None value`. The loop ended at its 5 s deadline before `session.failed` and `run.terminal` arrived. - Run 41892735530 (master): the same test failed in the same way. - Runs 41718881566 and 41670873220: `durable_descendant_lineage_survives_capacity_and_provider_restoration` failed with "capacity exhaustion must become a specific durable recovery reason". Its third phase waited only 5 s. An earlier phase of this test already had a 30 s deadline with a comment about this contention. **What did you expect?** The tests wait until the fake provider produces the event. They fail only when the provider hangs. **Steps to reproduce** Run `cargo test --release --workspace` for the runner on a contended Windows x64 runner. The suite takes about 380 s there, and some tests run for more than 60 s. The failures are intermittent. ## What Changed - `tests/codex_provider.rs`: add `EVENT_WAIT_TIMEOUT` (120 s) and `event_deadline()`. Replace all 50 fixed 5 s and 30 s wait deadlines with `event_deadline()`. Each wait already stops when its condition holds, so passing tests do not get slower. - `rejected_declared_completion_stays_failed_when_the_provider_completes`: assert that the run reached `run.terminal`, and print the observed event types if it did not. Replace the bare `unwrap()` on `session.failed` with a message that names the scenario. - `src/bin/fake-codex-app-server.rs`: the two background gate threads used 5 s and 10 s bounds. A test releases a gate only after its own waits succeed, so these bounds had the same problem. They now use one `TEST_GATE_TIMEOUT` of 120 s. - No production code changes. The two remaining `from_secs(5)` values are the unused `max_runtime` in the test config and a Unix-only latency assertion. Both stay the same. ## Verification - `cargo test --release -p paperclip-runner-core --test codex_provider --locked`: 109 passed, 0 failed, 2 ignored. - I ran the two flaky tests 48 times, with 6 copies in parallel per round to load the machine. All 48 runs passed. - `cargo fmt --all --check`: clean. - `cargo clippy -p paperclip-runner-core --test codex_provider --bin fake-codex-app-server`: no new warnings in the changed lines. The lib target already has clippy findings from the local toolchain (Rust 1.97). They are not in the changed files, and CI does not run clippy. - I did not reproduce the original failure on macOS. The slow condition occurs only on the loaded Windows runner. The Windows CI job on this pull request is the main check. ## Risks - Low risk. The change affects only tests and the test fixture binary. - If a test hangs for real, it now fails after 120 s and not after 5 s. This makes a broken run slower, but the failure message stays the same. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1eff07f29e |
feat(task-chat): show Claude subagents as a collapsible tree in the runner turn (#19264)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task chat shows what an agent does in a run. The Paperclip Runner sends this activity as canonical provider events. > - Claude runs subagents through its Agent (Task) tool. A run that delegates showed a confusing burst of unrelated tool calls. > - The subagent data was already sent, but it was lost: ACPX dropped `_meta`, the hardened sidecar dropped `rawInput`, and async subagents report "launched" as the tool result. > - This pull request maps each Agent call to a canonical `delegation` event, links each child tool to its subagent, and shows subagents as a collapsible card or tree. > - The benefit is that operators can see what each subagent was asked, what it is doing now, which tools it used, and what it returned. ## Linked Issues or Issue Description Refs #19252 (the docked plan card used the same canonical-event pattern) Refs #19259 (ACPX `_meta` passthrough, merged). This PR builds on it. **What happened?** When a Claude agent used its Agent tool, the task chat showed one generic "Task" row and then the subagents' tool calls as flat rows in the parent turn. The subagent type, prompt, progress and result were not visible. **Expected behavior** Each subagent shows as its own card with its type, description, prompt, live progress summary, nested tool rows and result. Subagents that start together show as one tree with status dots. **Steps to reproduce** 1. Assign a Claude (Paperclip Runner, ACPX) agent a task that asks it to launch three Explore subagents in parallel. 2. Open the task while the run is live. 3. Before this change, the subagents' tool calls appear flat in the parent turn. ## What Changed - **claude-agent-acp patch:** Under the isolated runner context, the patch turns on the SDK option `agentProgressSummaries`. It forwards each subagent's `task_progress` summary and its terminal state and summary onto the spawning tool call as `_meta.claudeCode.subagentProgress` and `subagentState`. Native subagent sessions stay off, because ACPX cannot follow child ACP sessions. - **Runner:** The new file `claude-subagent-lineage.ts` tracks each Agent/Task call across ACP update deltas. It maps the call to `delegation.started|updated|completed` events and stamps child tools with `parentItemId`. An async subagent stays running after its "Async agent launched" result until its own state settles. - **Sidecar path:** `acpx-runtime-sidecar.ts` reads the lineage before it drops `_meta` and `rawInput`. Delegations go out as schema-checked rich events. Child tools carry `parentToolCallId`, which Rust (`acpx_event_payload.rs`, `provider_events.rs`) maps to `parentItemId`. - **Contracts:** The provider-event schema adds optional `prompt` and `result` to delegation children and optional `parentItemId` to tool executions. The protocol modules are regenerated. ACPX Claude now advertises the `delegation` event family. - **UI:** `native-run-events.ts` keeps delegation payloads and the tool parent link. `buildTurnTimelineRows` groups adjacent delegations into a new `subagent_group` row, nests child tool rows (plain and provider-activity shapes), and hides the spawning call's own tool row. The new `TaskChatSubagentCard.tsx` renders the card and tree. A new size token caps the result height. - **Storybook:** The new story `Tasks and conversations/…/Task chat/Subagents` has 7 states. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/cli src/drivers/acpx src/provider-events.test.ts src/claude-subagent-lineage.test.ts src/protocol` (2069 passed) - `cargo test --release --locked -p paperclip-runner-core --lib` in `packages/paperclip-runner/runner` (488 passed, including the new parent-link test) - `node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs` (48 passed) - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript`, including the new `subagent-timeline.test.ts` - `check:protocol-types`, `check:protocol-manifest`, `check:acpx-profiles` and `pnpm check:token-gates` pass. - **Live test:** A local test-drive instance ran a Claude Opus 5.5 agent on the Paperclip Runner (hardened sidecar path). The agent ran real tasks that launched three parallel Explore subagents. The live view showed "Running 3 subagents · 2 done" with a progress summary. The settled view showed "Ran 3 subagents", each with its prompt, nested tools and its own result text. ## Risks - `agentProgressSummaries` makes Claude Code generate a short progress summary for each running subagent about every 30 seconds. This adds a small model cost to runs that delegate. - The adapter patch reads `task_notification` and terminal `task_updated` messages. A future claude-agent-acp version can change these. The patch hunks then fail to apply, and the dependency bump must update them. - If a delegation event is lost, child tools stay at the top level as before. Nothing is hidden without a matching delegation. - Codex is not changed. Codex subagents need `multi_agent`, which Paperclip keeps off on purpose. Changing that is a separate policy decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser automation). It did the investigation, implementation, tests and live test. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d361f92071 |
feat(task-chat): show harness status notices as chips, out of the answer (#19267)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. The runner turns harness events from Codex and Claude into canonical provider events, and the task chat shows them. > - Harnesses report their own lifecycle: context compaction, retries, model fallback, and warnings. Paperclip lost or mangled these reports. > - Claude (claude-agent-acp 0.73) sends compaction, model-fallback and warning banners as ordinary message text, so "Compacting..." ended up inside the agent's answer and the issue comment. > - Codex sends `error` with the text in `error.message` and a `willRetry` flag. Paperclip read `params.message`, so every Codex error said "Provider notice", and every retry looked fatal. > - This pull request keeps these reports out of the answer and shows them as small status chips between the turn's activity. > - The benefit is that users can see why an agent paused, retried or forgot earlier context, and the answer text stays clean. ## Linked Issues or Issue Description **Subsystem affected** Native runner provider-event mapping (TypeScript driver, ACPX sidecar and Rust runner-core) and the task chat transcript. **Problem or motivation** Claude status banners are mixed into assistant answer text. Codex error notices lose their message and their retry state. The task chat hid all of these notices, and settled turns stripped every provider notice. **Proposed solution** Classify Claude banners as provider notices at the ACPX boundary. Map Codex `TurnError` fields and `willRetry` correctly. Render status notices as inline chips. A retry chip fades after later provider work. A final error shows the real provider message. **Alternatives considered** Patch claude-agent-acp to tag banners in `_meta`. ACPX keeps only `origin`, `kind` and `source` from `_meta`, and other sessions own the ACPX `_meta` passthrough work. Banner chunks already have no `messageId`, and model output always has one, so the classification needs no adapter patch. **Roadmap alignment** This makes existing harness signals visible. It adds no new protocol schema. It uses the existing `paperclip.provider.notice.v1` `category` and `details` fields. ## What Changed - Codex `error` notifications (TypeScript `provider-events.ts` and Rust `runner-core/src/provider_events.rs`): read `error.message`, map `willRetry: true` to a recoverable `provider_retry` notice, keep category `error` for a final error, and add the `CodexErrorInfo` kind, HTTP status, and the "Reconnecting... n/m" attempt count as notice details. Retries in one turn share one notice id. - New `claude-status-banners.ts`: classifies the claude-agent-acp compaction, model-fallback and SDK informational banners. A chunk must have no `messageId` and match a known banner exactly. - ACPX sidecar and the TypeScript ACPX driver emit these banners as `provider_notice` events instead of answer text. Compaction start and end share one notice key and carry a `Phase` detail. - Rust runner-core accepts an optional `noticeKey` and bounded `details` on ACPX `provider_notice` events. - UI: new `TaskChatStatusNotice` chip and `status-notice-presentation.ts`. Status notices become top-level timeline rows, render as chips wherever protocol cards render, and stay visible in settled (historical) turns. Generic warnings stay hidden as before. - New tokens `--tc-status-notice-warning-*` in `ui/src/index.css`. - Storybook story `Tasks and conversations/.../Task chat/Status notices` with eight states. - `doc/run-log-events.md`: describes the status notice categories. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript` - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/provider-events.test.ts src/drivers/acpx/claude-status-banners.test.ts src/drivers/acpx/codex-acpx-driver.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `cargo test -p paperclip-runner-core --lib provider_events` and `--test acpx_provider_events` - `pnpm check:token-gates` - Storybook: open `product-task-chat-status-notices--all-states-gallery`. - Live test-drive on an isolated worktree instance: - Codex (gpt-5.6-sol, app-server 0.160): a local pass-through proxy in front of the real OpenAI API cut the first two response streams. Codex ran its own retries. The chat showed an amber "Reconnecting" chip live, then "Recovered after retry" once work continued. With every stream cut, Codex made five retries and then sent a final error with the real message. - Claude (claude-haiku-5-5 through claude-agent-acp 0.73): a long reading task triggered Claude's own auto-compaction twice. The chat showed "Compacting context…" live and "Context compacted" afterwards. The final answer and the issue comment contained no compaction text. - Not verified live: model fallback and Claude informational banners. They need a model refusal or a blocking hook. Unit tests cover them. ## Risks - Low to medium. A Claude model message that is exactly a banner text and has no `messageId` would become a notice. Model output from claude-agent-acp always carries a `messageId`. - Older stored Codex `error` notices (category `error`, not recoverable) now render as a red row instead of being hidden. - Claude retries (`api_retry`) are still not visible. claude-agent-acp reports them only to clients that negotiate session-failure records, and ACPX drops `_meta`. This is a follow-up that depends on the ACPX `_meta` passthrough work. - On a failed Codex turn the thread collapses to a summary, so the red row does not show there. The existing task banner already shows the real provider message. - Some local suites failed on this machine. They do not touch the changed code: Cursor Cloud staging-custody tests, the ACPX Claude dependency-graph pin test (a timeout), and the Rust `task_memory` monitor tests. CI is the reference for these. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution and browser automation. Default context window. Extended reasoning was not enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs paperclipai/paperclip#19252 (plan card precedent). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
088f85c5fe |
fix: rename the managed environment to Paperclip Sandbox (#19296)
## Thinking Path > - Paperclip manages AI agents and their execution environments. > - Managed installations supply a default sandbox through managed configuration. > - Most selectors show the configured environment name. > - A fallback label, two errors, and examples still use Paperclip Computer. > - This pull request changes that stock copy to Paperclip Sandbox. > - Existing and custom environment names continue to come from the stored row. ## Linked Issues or Issue Description **What existing behavior does this improve?** The managed environment label in agent settings and the errors shown when the managed sandbox is unavailable. **Subsystem affected** The agent configuration UI, runner selection, and environment reconciliation tests. **Current behavior** Fallback copy and current examples call the managed sandbox Paperclip Computer. The actual environment name is already configurable. **Proposed behavior** Use Paperclip Sandbox for stock copy and current examples. Keep configured names unchanged in selectors, including legacy names and custom names. **Reason and benefit** The name identifies this environment as a managed sandbox and removes ambiguity. **Breaking changes** The displayed fallback and error text change. Error codes, execution behavior, and environment identity remain the same. No migration is needed. **Additional context** No duplicate rename issue or pull request was found. This is a copy update, with no new roadmap feature. ## What Changed - Update the managed fallback label and two unavailable-sandbox error messages. - Update current test and Storybook fixtures without changing story IDs or behavior. - Test configured names, fallback copy, and the unavailable-sandbox message. - Add a PostgreSQL regression test for a rename that retains the environment ID, instance default, agent binding, environment variables, and lease. ## Verification - `pnpm -r typecheck` passed. - `pnpm build` passed. - Focused coverage: 956 cases across nine changed suites. The new fixture initially tried to insert an existing singleton; its corrected upsert passed with all 52 environment-service cases. The updated runner suite also passed all 243 cases. - The two UI suites passed all 189 cases, including configured legacy/custom names and the missing-default fallback. - `pnpm check:token-gates` passed. - `node scripts/storybook-catalog.mjs --check` passed. The catalog is structural evidence only. - `node --test scripts/check-maps.test.mjs scripts/storybook-catalog.test.mjs scripts/storybook-feature-index.test.mjs` passed. - Full `pnpm test:run` is in progress. No live provider or browser journey is claimed by these tests. - CI passed on the current head after one server-shard rerun. The first attempt hit a 10 ms deadline assertion in the unchanged status-card timing test; its isolated local run and the complete rerun shard passed. The original failure and rerun are retained in the PR comment. - Reviewed the diff for secrets, personal data, and run artifacts. ## Risks Low risk: runtime edits change three display strings. Managed reconciliation is unchanged. The regression test exercises the existing rename against PostgreSQL. The configured name still controls normal display, so this code change alone does not rename existing database rows. A managed configuration update uses the existing reconciliation path. Roll back the copy by reverting this commit. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, xhigh reasoning, with repository tools and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, version, and capabilities - [x] I have checked ROADMAP.md; this copy update does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue in the PR with the enhancement template fields - [x] No internal or instance-local issue references - [x] The branch name describes the change and contains no internal ticket ID - [x] Focused tests pass locally - [x] Relevant tests were added or updated - [x] Relevant copy and examples were updated; no API or schema documentation changed - [x] Risks are documented - [x] All CI gates passed on the current head, including the server 11/12 rerun and aggregate verify - [ ] Full local test suite completed (still running; six chat shards passed) - [x] Greptile is 5/5 on the current head with no unresolved findings - [x] I will address all reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
cb0b7525e0 |
feat(task-chat): warn at the top of a run when an MCP server failed to start (#19289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each agent run shows its activity in the task chat. The Paperclip runner turns harness events into that activity. > - Codex reports `mcpServer/startupStatus/updated` when an MCP server fails to start. The runner dropped this event at the notification allowlist and in the Rust normalizer. > - When an MCP server fails, the agent runs without its tools. The agent then says "I could not reach the server", and the user cannot see why. > - This pull request maps a failed MCP start to a user-facing provider notice and shows it as one warning line at the top of the run. > - The benefit is that the cause of a "cannot reach my tools" answer is visible next to that answer. ## Linked Issues or Issue Description Refs #19252 (the plan card, which set the pattern for promoting a dropped harness signal to a dedicated task-chat surface). **Problem description** Codex sends `mcpServer/startupStatus/updated` (`name`, `status`, `error`, `failureReason`) during `thread/start`. The TypeScript allowlist (`codex-thread-normalization.ts`) and the Rust normalizer (`runner-core/src/provider_events.rs`) both dropped it. A run with a broken MCP server showed no sign of the failure. The agent only said that it could not reach the tools. **Proposed solution** Map `status: "failed"` to `provider.notice.recorded` with `category: "mcp_server_startup_failed"`, `severity: "warning"`, `scope: "session"` and `userActionable: true`. Show these notices as one run-health line under the runner header: "github MCP server failed to start: No such file or directory (os error 2)". **Alternatives considered** An inline notice row in the timeline. It scrolls away and sits among tool rows. A run-level line next to the header is easier to connect to the agent's answer. ## What Changed - Runner (Rust, production Codex path): `normalize_codex_notification` maps a failed `mcpServer/startupStatus/updated` to a `provider.notice.recorded` notice. `starting`, `ready` and `cancelled` stay out of the transcript. The new helper removes the Codex wrapper text ("MCP client for \`x\` failed to start: MCP startup failed:") so that the line shows the real reason. `reauthenticationRequired` becomes "needs re-authentication". - Runner (TypeScript): the same mapping in `canonicalProviderEventsFromCodex`, and the method is added to `isSupportedCodexNotificationMethod`. - Server: the `view=conversation` run-event query keeps `mcp_server_startup_failed` notices. A settled run can then still show the warning. Other notices stay out of that view. - UI: new `TaskChatRunHealthBanner`. `partitionRunHealthItems` takes MCP failures out of the run's items, with one line per server where the latest report wins. `TaskChatRunnerTurn` (live runs) and `TaskChatTurn` (settled, historical and merged turns) show it under the header. `coalesceSettledTurns` merges the failures when it merges turns. - Storybook: `Tasks and conversations/…/Task chat/Run health` shows one failure, several failures, a finished run and a healthy run, all through the production runner turn. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/provider-events.test.ts src/drivers/codex src/protocol`: 508 tests pass. - `cargo test --release -p paperclip-runner-core --lib -- provider_events`: 20 tests pass. - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript src/components/TaskChatThread.test.tsx`: 1340 tests pass, including the new `TaskChatRunHealthBanner.test.tsx` (persisted event to rendered banner, deduplication, settled historical turn, merged turns). - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/task-cold-open.test.ts`: the conversation view keeps the MCP notice and still excludes other notices. - `pnpm check:token-gates` is clean. `node scripts/typecheck-low-memory.mjs` passes. - Live test on a local instance with Codex 0.160: a Codex agent with a preconfigured Codex home whose `config.toml` has `[mcp_servers.github] command = "github-mcp-server-not-installed"`. Codex reported the failure during `thread/start`. The persisted event had the summary "github MCP server failed to start: No such file or directory (os error 2)". The banner showed while the run was live. It also showed on the settled turn after "More run output", above the agent's answer that it could not reach the GitHub MCP server. ## Risks - Low risk. The change adds one notice category. It does not change how existing notices map or render. - A settled run whose transcript is deferred on a cold open (a successful run with a posted agent comment) shows the warning only after "More run output". This is the same as all other run activity under the current compact history. - Claude does not report MCP startup failures over ACP. `claude-agent-acp` reads MCP status only for OAuth. This change is therefore Codex-only. - Provisional task titles from provider session names are not part of this change. Codex app-server does not generate thread names (only its TUI does). Claude generates its title after the turn ends, and ACPX drops the title text. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (shell, file edits, browser automation for Storybook and the live test). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b400f76647 |
feat(ui): submit task composers with Enter by default (#19272)
Use Enter to submit task replies and new tasks, with Shift+Enter for a new line. Add an experimental preference to require Command/Ctrl+Enter. Preserve autocomplete selection and IME confirmation, and retain the modifier shortcut until saved settings load. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c6996799c4 |
feat(task-chat): show an inline diff for every file edit (#19269)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat shows what an agent did during a run, including each tool call > - Codex and Claude both report the patch for every file edit, but Paperclip dropped it. Claude's ACP `diff` content was never read. The Codex `fileChange` item became a generic item with no patch. Codex `turn/diff/updated` had no native-runner mapping > - A reviewer could not see what an agent changed without opening the workspace > - This pull request carries a bounded patch on each edit call and shows it in the edit row. Each turn that changed files ends with a "Changed in this turn" card > - The benefit is that a reviewer can check every change in the conversation, with the same view for Codex and Claude ## Linked Issues or Issue Description Refs #19252 (precedent: canonical `plan.updated` events as a docked plan card). **Subsystem affected** Paperclip runner (Rust and TypeScript canonical event mapping), PRP provider-event schema, and the task-chat UI. **Problem or motivation** Edit rows say "Edited a file" with no content. Claude Edit/Write `diff` content is lost after the ACPX sidecar. The Codex native runner drops `fileChange.changes` and `turn/diff/updated`. The existing workspace-changes card showed a raw text patch in a dialog, and only on the in-process Codex path. **Proposed solution** Add an optional, bounded `fileChanges` list to `paperclip.tool.execution.v1`. Fill it from Codex `fileChange` items and Claude ACP `diff` content. Map Codex `turn/diff/updated` to `workspace.change.updated` on the native runner. In the UI, show `+N −M` on each edit row, open a compact colored diff, and end the turn with the change set. **Alternatives considered** A runner-side git diff at turn end. It needs workspace access and extra cost on every turn. The harness patches are already available. For Claude, the UI builds the turn card from the edit rows instead of new per-turn runner state. ## What Changed - Protocol: optional `fileChanges[]` on `toolExecution` in `provider-event.schema.json`, with the regenerated schema bundle, validators and manifest. Paths are workspace-relative. A patch is at most 32 KiB, and a call has at most 32 files. - Runner (Rust): new `tool_file_changes.rs`. A Codex `fileChange` item now also emits `tool.execution.*` with `fileChanges`; the native item event stays for the recovery and progress projections. `turn/diff/updated` becomes `workspace.change.updated`. `normalize_acpx_tool_call` re-validates the sidecar's `fileChanges`. `normalize_codex_notification_in` resolves absolute paths against the session cwd. - Runner (Rust): `has_bounded_rich_display_shape` accepts schema-valid workspace diffs. Without this change, durable storage cut each patch string to a 4 KiB preview. Patches use the uncapped secret redactor. - Runner (TS): new `tool-file-changes.ts`. The ACPX sidecar computes `fileChanges` from ACP `diff` content with the working directory. The in-process Codex and ACPX mappers do the same, with an optional `context` argument. - UI: new `TaskChatDiffView` (colored rows, hunk labels, a height cap, "Show N more lines"). Edit rows show counts and the diff. Collapsed activity groups show edit totals. The workspace card is now "Changed in this turn", with in-place per-file diffs, and it sits at the end of the turn. When a harness sends no whole-turn diff (Claude), the card is built from the edit rows. A change set is marked complete when its run ends. - UI: `native-run-events.ts` maps `workspace.change.updated` and attaches each edit's latest patch to its row (the `diff` transcript entries now carry `toolUseId`). - Design tokens: `--diff-add-*`, `--diff-remove-*` and `--sz-edit-diff-max-h` come from the status icon hues. - Storybook: `Tasks/Edit diffs` story. Docs: `doc/run-log-events.md`. ## Verification - Real runs on a local instance (`test-drive`, native runner): - Codex (`gpt-5.6-sol`): the "Applied a patch" rows show `+9` and `+35 −2` with their diffs. The turn ends with "Changed in this turn", 2 files, `+42`, from `turn/diff/updated`. - Claude (`claude-opus-5-5` via ACPX): the Write and Edit rows show `+5`, `+14 −1` and `+9` with correct hunks. The turn card is built from the edit rows. - Codex model note: with `gpt-5.4`, pinned Codex 0.160 reports "model metadata not found". The model then wrote files through the shell, so no `fileChange` items occurred. - `pnpm check:token-gates`: all gates clean. - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript src/adapters`: 1073 passed. - `packages/paperclip-runner`: `vitest run src/provider-events.test.ts src/tool-file-changes.test.ts src/cli/acpx-runtime-sidecar.test.ts` passed. `check:protocol-types` and `check:protocol-manifest` match. - `cargo test -p paperclip-runner-core --lib provider_events tool_file_changes`: passed. `cargo fmt --check`: clean. - Storybook: open `Tasks and conversations/…/Tasks/Edit diffs`. ## Risks - Larger payloads: an edit call can carry up to 96 KiB of patch in Rust, and the sidecar keeps 48 KiB under its 128 KiB frame. Above the budget, a file keeps its counts and drops its patch (`diffTruncated`). - Redaction: each patch goes through the shared secret redactor. Before this change, a 4 KiB preview cap also applied. - Claude turn card: it joins each edit's patch in order. It is not a net diff. For a file the turn created, the counts are net and exact. For an existing file, the counts are summed over its edits. - The workspace card moves to the end of the turn. One adapter test changed to match. - Local only: four Linux cgroup `task_memory` Rust tests fail on macOS. They do not touch these modules. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, browser and file editing) and a long context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
29afc633f6 |
feat(runner): forward ACP _meta from ACPX as a bounded copy (#19259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner runs Claude through ACPX and maps ACP updates into canonical run events. > - Claude sends useful data in the ACP `_meta` field: the real tool name, the parent tool of a subagent call, the terminal exit code, and the rate-limit state. > - ACPX 0.13.1 drops all `_meta` before the runner sees it. > - Three planned features need this data: richer tool rows, a rate-limit gauge, and a subagent tree. > - This pull request forwards `_meta` from ACPX and bounds it once in the runner. > - The benefit is that each feature can read the data from one safe, shared place. ## Linked Issues or Issue Description **Problem or motivation** ACPX keeps only the title, status, kind, locations, raw input, raw output and content of an ACP tool update. It drops `_meta`. Claude's tool name, `parentToolUseId`, `terminal_exit` and `_claude/rateLimit` never reach the runner. **Proposed solution** Forward `_meta` from ACPX on tool and status events. Copy it into a bounded, JSON-safe object in the runner before any mapper reads it. **Alternatives considered** Read the full ACP notification through `onSessionNotification` and join it to runtime events by tool call ID. That needs a side table and depends on callback order. Forwarding on the event is simpler and keeps the data with the event. ## What Changed - `patches/acpx@0.13.1.patch`: ACPX now sets `_meta` on `tool_call` events and on status events (`usage_update`, `current_mode_update`, `config_option_update`, `session_info_update`). The type declarations document the field as unvalidated producer data. - `packages/paperclip-runner/src/provider-events.ts`: new `boundedAcpMeta()`. `createAcpxToolEventNormalizer` applies it to every runtime event. Limits: depth 6, 64 entries per object or array, 4 KB per string, about 16 KB in total. It drops `__proto__`, `constructor` and `prototype` keys, functions and non-finite numbers. - The hardened ACPX sidecar path keeps its explicit field allowlist. It does not forward `_meta` yet. A feature that needs a field there must add it to that allowlist. - Tests for the passthrough and for each bound. ## Verification - `cd packages/paperclip-runner && npx vitest run src/provider-events.test.ts` - The canonical mappers do not read `_meta` yet, so run events do not change. Follow-up pull requests use it. ## Risks - Low risk. The new field is additive and the mappers ignore it today. - The lockfile patch hash for `acpx@0.13.1` changes. A fresh `pnpm install` applies the new patch. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d8c47159d4 |
feat(task-chat): show Codex reasoning summaries as headline bullets (#19271)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat shows what an agent does while it runs, including a "Thinking" row for provider reasoning > - Real Codex runs produced reasoning items with no text > - So the Thinking row only says "Thinking" and never tells the user what the agent works on > - This pull request asks Codex for its own reasoning summaries, keeps every summary part, and shows the parts as bullets > - The benefit is that a user can see what the agent is thinking about, and a collapsed live run shows the newest headline as its status line ## Linked Issues or Issue Description Refs #19252 (same pattern: show a harness signal that Paperclip already receives) **Subsystem affected** Native runner (Rust runnerd Codex provider and event normalization, JS Codex driver) and the task chat UI (Thinking row and activity rows). **Problem or motivation** Codex 0.160 resolves an unset reasoning summary from the model, and current models resolve it to `"none"`. The rollout of a real gpt-5.6-sol run records `turn_context.summary = "none"`, and all of its reasoning items have `summary: []`. A local A/B with `codex exec` proved the cause: `model_reasoning_summary=none` gives no reasoning text, and `auto` gives headlines. Also, runnerd dropped `item/reasoning/*` notifications and ignored the reasoning item's `summary[]`. **Proposed solution** Send `summary: "auto"` on Codex `turn/start`. Map `item/reasoning/summaryTextDelta` and `summaryPartAdded` to reasoning deltas, with a blank line before each later part. Use the joined `summary[]` as the completed item text. In the UI, split the provider text at headline lines and show one bullet per part. **Alternatives considered** Use `summary: "detailed"`. That gives longer text, but `auto` lets the provider choose the best summarizer. **Roadmap alignment** This makes existing provider signals visible. It adds no new protocol event type or data model. **Additional context** Research note: overlooked harness signals, item 2. Claude is out of scope here. A live Claude run showed thinking blocks with signatures and empty text: current models default `thinking.display` to `"omitted"`. Claude Code ignores the `showThinkingSummaries` setting in SDK mode, and ACPX does not forward a `thinking` option, so the fix needs a claude-agent-acp patch hunk. That patch is also changed by #19264, so the Claude change follows after #19264 merges. The UI in this PR already shows prose-only (Claude) summaries. ## What Changed - Rust runnerd (`codex_provider.rs`): send `summary: "auto"` on every Codex `turn/start`. - Rust runnerd (`provider_events.rs`): map `item/reasoning/summaryTextDelta` and later `summaryPartAdded` to `item.delta` with `kind: "reasoning"` and `channel: "summary"`. Use the joined `summary[]` as a completed reasoning item's text. - JS Codex driver: send `summary: "auto"` on direct app-server transports, map `summaryPartAdded`, and read `summary[]` in `itemText`. - UI: new `reasoning-summary.ts` splits provider text into headline sections, and `TaskChatReasoningSummary` shows one bullet per part. The Thinking row and the activity row use the newest headline as their one-line status. Prose-only summaries (Claude) stay as prose. - Storybook: new "Task chat reasoning summaries" story with live, expanded, finished and Claude states. - Tests: Rust normalizer test, JS driver test, and splitter unit tests. ## Verification - `cargo test -p paperclip-runner-core` (provider events and the shared identity golden). - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/codex` - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript` - `pnpm check:token-gates` - Live test on a local worktree instance with a real Codex agent (gpt-5.6-sol). The Codex rollout now records `summary: "auto"`. Run events contain `**Assessing transaction boundary issues**`, a part break, and `**Planning atomic claim fix and verification**`. The task chat shows "Thought · Planning atomic claim fix and verification", and the expanded row lists both headlines as bullets. - Storybook: open "Tasks and conversations / Task creation and lifecycle / Tasks / Task chat reasoning summaries". - Claude was not changed. A live Claude run confirmed the cause for the follow-up (empty, signed thinking blocks). ## Risks - Codex now writes reasoning summaries, which adds a small number of output tokens per turn. This is the same cost as Codex CLI with its normal default. - The UI treats a line that is only bold text as a headline. A prose line that is only bold text also starts a new bullet. This is display only. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution and browser automation. Default context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
036b1860c3 |
fix(runner): bind Cursor sidecars to the runner package, not the workspace root (#19255)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner starts ACPX providers through a sidecar. runnerd
tells the sidecar which package owns the provider assets.
> - Cursor, Copilot, Pi, Kimi and Antigravity use pinned native
closures. Their asset guard requires the bound manifest to be the
`package.json` directly inside the bound root.
> - In a local source build, runnerd bound Cursor to the pnpm workspace
root, with the runner manifest one level down. The guard threw before it
checked the Cursor closure.
> - Every Cursor run on a local source instance failed with
`admission_installation`. The sidecar redacts the message, so the cause
was not visible.
> - This pull request gives Cursor the same runner-package authority as
the other native-asset providers.
> - The benefit is that Cursor runs start on local source builds.
## Linked Issues or Issue Description
**What happened?**
A `paperclip_runner` agent with `provider: "acpx"` and `acpxAgent:
"cursor"` failed at every start on a local macOS worktree instance. The
run log showed `ACPX sidecar command session.open was rejected
(retryable=false, classification=admission_installation)`.
**Expected behavior**
The run starts, because the pinned Cursor runtime cache matches
`cursor-distributions.json`.
**Steps to reproduce**
1. Run `pnpm dev` from a source checkout, with the Cursor runtime set up
in `~/.paperclip/runtimes/cursor`.
2. Create a native runner agent with `acpxAgent: "cursor"` and a bound
`CURSOR_AUTH_TOKEN`.
3. Start a run. It fails at `session.open`.
**Paperclip version or commit**
`master` at
|
||
|
Nicky LeachandPaperclip
|
3d0d178ff7 |
Reconcile qualified ACPX predecessors through ordinary recovery (#19153)
## Thinking Path > - Paperclip runs agent work through a controller and a Runner. > - A completed provider session can outlive an abrupt Runner exit. > - The next owner must reconcile the retained source before it attaches a new run. > - The ordinary recovery foundation in #19132 first supports Codex App Server. > - Qualified ACPX providers need their own saved and live consistency checks. > - This pull request adds those checks and a separate provider attestation. > - The benefit is session continuity through existing recovery and attachment. ## Linked Issues or Issue Description Builds on #19132, which has merged. This PR targets `master` and contains only the ACPX changes. **What existing behavior does this improve?** Recovery of a completed ACPX session after the Runner and provider stop. **Current behavior** Ordinary ACPX recovery and attachment preserve a qualified session. A completed predecessor also needs trusted admission by its new owner before restoration. Codex checks cannot attest ACPX state. **Proposed behavior** Reuse ordinary exact-source recovery. Compare live restored state with saved dormant state when no uncertain effect explains new work. Require the selected driver's separate ACPX prelaunch attestation. Settle retained events before authenticated target attachment. ## What Changed - Preserve baseline prepared cleanup and ordinary continuation. New predecessor admission excludes prepared provider state; snapshots and cleanup do not restart it. - Observe existing recovery command facts. Keep ordinary active same-run behavior and existing profile, workspace, model, permission, session and catalog checks. - Verify the restored qualified session's identity, turn, goal and outstanding requests when saved state was dormant. Failed verification reaps the owned process group. - Validate scoped goal updates arriving during the final status snapshot after the authoritative goal reply. Active or unknown updates fail admission; earlier and benign queued events remain available to ordinary polling. - Recognize the existing inactive `limited`, `usage_limited` and `budget_limited` goal statuses, so an unexpected active goal after restore cannot bypass live verification. Treat malformed nested working-state metadata as unproven while keeping missing-field compatibility. - Keep existing ordinary recovery of qualified session-open and suspended ACPX sessions. The new integrity check does not expand prepared restoration. - Advertise `acpxRecoveryIntegrityVersion: 1`, separate from Codex integrity. No native source mode or CLI remains. - Require `acpxSourceRecoveryVersion: 1` from the selected trusted execution driver for a stopped predecessor. Codex capability alone cannot grant it. - Reuse dynamic authenticated owner admission, retained source accounting, and existing ordinary ACPX attachment. ## Verification Current head `b294d8ae88d0` is one commit on `master` at `e80d745060fd`, the merge of #19132. The ten-file diff is byte-for-byte identical to the reviewed ACPX diff against its former base. The guide shows where `paperclip-runnerd --build-metadata` emits `acpxRecoveryIntegrityVersion: 1` and how host admission uses a separate capability. [ACPX recovery guide](https://github.com/paperclipai/paperclip/blob/b294d8ae88d019ef02a942e240bc5c14287239e0/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md#recovering-acpx-between-turns) The rebased head passed all 14 ACPX process cases, three native snapshot checks, Runner typecheck and the binary metadata check locally. GitHub CI, Docker and Windows passed on their first attempts. Standard Greptile review 12 scored the rebased commit 5/5 with zero unresolved threads. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42063188178) · [Standard review](https://github.com/paperclipai/paperclip/pull/19153#issuecomment-6646809075) The preceding code commit `08df48de20d613efcd861c278a7f9929c21d6670` includes the reviewed parent `c09ebb4a2a6c140521698cf9fd681de67cfb9745`. Paid review 9 scores this exact head 5/5 with no unresolved findings (summary updated 2026-11-05 09:13:11 UTC after the 09:07:15 UTC request). Standard review 8 also scored this head 5/5. [PR workflow 41995103666](https://github.com/paperclipai/paperclip/actions/runs/41995103666), [Runner Windows 41995103419](https://github.com/paperclipai/paperclip/actions/runs/41995103419), and [Docker Runner 41995103366](https://github.com/paperclipai/paperclip/actions/runs/41995103366) passed on this exact head. The initial main attempt lost self-hosted runners: the Runner package log records a shutdown signal, and the other affected jobs report lost communication or cancellation. Only unsuccessful jobs were retried; their successful retry and aggregate gate are green. No source, assertion or timeout change was made for this interruption. - Combined `runnerd-acpx-source-reconciliation.test.ts` and `runnerd-source-reconciliation.test.ts`: 25 real native/fake-provider process cases passed. ACPX contributes 14 cases, including the final-snapshot goal race, process-group cleanup, same-PID adoption, exact provider identities, retained accounting, limited/paused goals and ordinary active same-run recovery. - The maintained fake sidecar emits a real `runtime.goal` frame between the dormant goal reply and final snapshot response. Before the fix, recovery succeeded. Afterward it refuses admission, reaps the owned sidecar/provider group, and records only the original provider turn. - Three native wire tests passed. They cover active updates with low/equal/missing revisions, foreign run/session identity, and unchanged ordered delivery of earlier history plus benign new events. The queue is inspected without consuming frames. - Both parent Codex final-read tests passed in the combined checkout: active-goal refusal/reap and safe paused-goal delivery. - The real native backend completes A, settles its journals, kills Runner and provider, recovers the same ACPX session, adopts the replacement Runner after controller restart, attaches B and completes a second turn, then uses ordinary warm attachment for C and completes a third successful turn. Record, backend and agent session identities remain exact; ordinary ACPX attachment may restart its sidecar. - Provider compatibility checks also passed while preparing this stack: 30 permission ordering/forwarding cases, 19 factory cases, the native inactive-goal classification unit, and six native warm/suspend integration cases. - Combined offline Runner/fake-provider build, Runner `tsc --noEmit`, `cargo fmt --all -- --check`, parent-relative `git diff --check`, and added-line secret/personal-data scan passed. - Only focused local checks ran. Full suites and platform checks run in CI. ## Risks - Fake providers prove protocol behavior, not vendor restore inertness or all vendor background work. Real provider qualification remains required. - The trusted process owner must reject unsafe completed predecessors before launch. Ordinary same-run active-goal recovery remains supported. - Probes use the existing bounded sidecar budget. A timeout can terminate the sidecar through existing transport cleanup. Null goal projections remain supported. - Ordinary ACPX attachment can restart the sidecar. Record, backend and agent session identities remain exact; the Runner PID remains the same. - This draft adds no deployment, release pin change or parking hook. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with local code and command tools. ## Checklist - [x] I have included a thinking path and specified the model - [x] I have checked ROADMAP.md and confirmed this does not duplicate planned work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have not included internal or instance-local references - [x] My branch name describes the change - [x] Focused local tests pass - [x] Tests and documentation cover the changes - [x] Risks are documented - [x] All Paperclip CI gates are green at the revised head - [x] Greptile is 5/5 with no unresolved comments at the revised head - [x] I will address review findings before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
b0fa2ef9bf |
feat(task-chat): show web searches, model switches, waits and verification checks as dedicated rows (#19262)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. The runner turns harness activity into typed canonical events such as `research.*`, `model.route.changed`, `wait.*` and the run's own verification checks. > - The task chat flattened these events into generic tool rows. A web search showed as "Ran Research eventType: research.started, summary: …". The run's verification checks were dropped completely. > - Live tests found a second break: the Rust runner core forwarded Codex `webSearch` items without their query, action or results. The canonical `research.*` events therefore had `query: null`. > - This pull request keeps the full payload for these families (as #19252 did for plans), adds small dedicated rows, and keeps the web-search fields through the durable runner. > - The benefit is that users see what the agent searched for, which model actually ran, how long a wait lasts, and whether each check passed, without opening the run log. ## Linked Issues or Issue Description Refs #19252 (same pattern for the plan card) **Subsystem affected** Task chat activity rows (UI), the native run-event transcript mapping, and the Rust runner core Codex normalizer. **Problem or motivation** Provider activity that the runner already maps to canonical events reaches the task chat as unreadable generic rows. Verification checks never appear. For Codex, the web-search query and action are lost before they reach the canonical event. **Proposed solution** Keep `research`, `model_identity`, `artifact`, `review` and `wait` events as `provider_activity` entries with their full payload. Project the runner's `verification` items as a new `verification` family. Render dedicated rows: a web search with its quoted query and source-domain chips, a model switch as `from → to · reason`, a wait with a live countdown, and green/red check lines. Keep bounded web-search and sleep fields in the Rust normalizer. **Alternatives considered** A separate "Checks" block beside the final answer. We kept the checks in the activity group, because the group summary already leads with "Check failed" when a check fails. ## What Changed - `ui/src/components/transcript/native-run-events.ts`: keep research, model, artifact, review and wait events as `provider_activity` with the full payload. Map `item.completed` items of kind `verification` to the new `verification` family. Context, hook, safety and delegation stay on the generic path, because other work items own them. - `packages/adapter-utils/src/types.ts`, `ui/src/components/task-chat/task-chat-model.ts`, `ui/src/lib/issue-chat-messages.ts`: add the `verification` provider-activity family. - `ui/src/components/task-chat/task-chat-activity-presentation.ts`: add labels: "Searched the web “query”", "Opened a web page host/path", "Switched models a → b · reason", "Waiting 30s" / "Waited 30s", and "Check passed" / "Check failed" / "Check not run". - `ui/src/components/task-chat/TaskChatRunnerActivityGroup.tsx` and `TaskChatProtocolActivityRow.tsx`: show up to three source-domain chips under a finished search, a live countdown for a running wait, and a check line with a tinted icon and its detail. - `ui/src/components/task-chat/completed-activity-summary.ts`: the group summary leads with "Check failed" when a check fails, and says "Switched models" for a reroute. - `ui/src/components/task-chat/transcript-adapter.ts`: verification detail keys, and a `startedAt` timestamp that stays across lifecycle merges (used by the countdown). - `packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs`: keep bounded `query`, `action` and http(s) `results` for Codex `webSearch` items, and `durationMs` for `sleep` items. Other item types do not change. - `ui/storybook/stories/tasks/task-chat-provider-activity.stories.tsx`: new story. Its fixtures are persisted run events that go through the production transcript path. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/transcript src/components/task-chat src/components/TaskChatThread.test.tsx src/adapters/paperclip-runner src/lib`: 275 files and 3669 tests pass. - `cargo test -p paperclip-runner-core`: all tests pass, including the new `keeps_bounded_web_search_and_sleep_identity`. - UI typecheck is clean. `pnpm check:token-gates` is clean. `cargo fmt --check` is clean. - Storybook: open "Tasks and conversations / … / Task chat / Provider activity". It shows a settled run, a live search, a live wait countdown and a narrow layout. - Live test on a local worktree instance with a real Codex agent (gpt-5.6-sol, Codex app-server 0.160). The task asked for a web search, a parser with tests, a pause, and two checks. The task chat showed "Searched the web “ISO 8601 week date first week Thursday conversion calendar date”" and two green check rows ("node --test week-date.test.js · 8 tests passed; 0 failed.", "node --check week-date.js"). The group summary read "Ran checks". Before the runner fix, the same search had `query: null`. Not verified live: - The source-domain chips. Codex 0.160 sends `results: null`, because it keeps search results on the server. The chips appear only when a provider reports sources. Unit tests and the story cover them. - The wait countdown and the model-switch row. In the live run Codex paused with a shell `sleep`, not a native `sleep` item, and no reroute occurred. Unit tests and the story cover both. - Claude. Its WebSearch and WebFetch calls arrive as generic tool executions, because ACPX drops the tool metadata. They show as tool rows, as before. The ACPX `_meta` work is separate. ## Risks - Low risk. The change only affects how canonical events display. The generic path stays for the families that this PR does not change. - The Rust normalizer now persists the web-search query and the source URLs/titles in the run events. This data was already intended for the canonical `research.v1` event, and it is bounded (4,000 characters per text field, 64 results, http(s) URLs only). - A search that Codex reports with `action: other` and no query shows as "Searched the web" with no subject. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, browser automation) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b1d00eeca0 |
fix(evals): derive the controlled Runner eval policy from committed defaults (#19253)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification runs real providers against 41 cases and reports them in TestOps. > - Controlled mode (#17889) adds a spend cap, private evidence storage, and cost and trace accounting. > - Operators must hand-maintain one large JSON variable for each campaign size, price review, and lockfile change. > - A one-cell policy blocked the first 41-case run, and its prices expired after 24 hours. > - This pull request derives that policy from committed defaults and the selected cases. > - The benefit is that a full run needs no policy edit, and all three safety properties stay. ## Linked Issues or Issue Description Refs #17889, #19011, #19070. **What existing behavior does this improve?** Controlled Runner protocol campaigns read `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON`. That variable holds the cell limit, a price table with a review expiry, a pinned lockfile digest, and self-hashed bound proofs. **Current behavior** A campaign with more cells than `maxCells` fails before provider work. Expired prices also fail before provider work. Any change to the lockfile needs a new pin. A one-cell policy made all 41 TestOps cases fail as infrastructure, so an operator had to edit the variable by hand before the full run. **Proposed behavior** The workflow derives the policy at preparation time. The cell count comes from the selected catalog. Prices come from a committed table with a version and a source URL. The controller installs the committed frozen lockfile. One optional variable, `RUNNER_PROTOCOL_EVAL_MAX_USD`, can override the committed $400 spend cap. **Reason and benefit** Each piece that was hand-maintained duplicated a check that already exists. The manifest digest binds every allocation. The frozen install binds dependencies to the reviewed commit. Under model-request readiness, the self-hashed bounds never changed admission or accounting. **Breaking changes** The old variable is no longer read. It can stay set or be deleted. Retained manifests that include the old proofs still replay. ## What Changed - Add `packages/paperclip-runner/scripts/controlled-evals/hosted-defaults.json`. It holds the spend cap, duration, route, Claude Sonnet 5 prices, allocation, overhead reservation and trace limits. - Add `hostedPolicy()` in `hosted.mjs`. It derives `maxCells` from the selection. It gives each campaign a validity window from preparation to one hour after its deadline. It applies the optional spend cap override. - Remove `capacityBound` and the policy self-hash from hosted preparation. `manifest.mjs` needs model `provisionalBound` only when readiness is not `model_requests`. - Workflow: stop reading the policy variable. The controller build installs the committed `pnpm-lock.yaml` with `--frozen-lockfile` and fails on any lockfile drift. Preparation checks that the runtime receipt lockfile digest equals the committed lockfile. The catalog limit is now 1 to 512 selected cells. - Keep everything else the same: the spend-cap check before provider work, private S3 retention of the manifest and derived `hosted-policy.json`, receipts, tracing, first-attempt and zero-retry gates, and paid-environment approvals. - Update `hosted.test.mjs`, `portable.test.mjs`, `runner-protocol-eval-workflow-security.test.mjs`, and `docs/runner-protocol-live-evals.md`. ## Verification - `node --test scripts/controlled-evals/hosted.test.mjs`: 8 pass. The new tests show that `maxCells` equals the selection and that the validity window covers the deadline with no committed expiry. They also show that the spend cap override applies. With the committed defaults, 41 cells validate with $339 reserved, and 49 cells fail on the $400 budget. - Negative controls still fail: drift in selection, source or attempt; a bad lock digest; missing tracing; a duration over 4 hours; a malformed or too-low spend cap; and zero cells. - `node --test scripts/runner-protocol-eval-workflow-security.test.mjs`: 20 pass. Controlled admission now passes whether the old variable is absent, still set, or malformed. A real git checkout with a modified `pnpm-lock.yaml` fails the controller guard before install. - `node --test scripts/controlled-evals/controller.test.mjs scripts/controlled-evals/portable.test.mjs`: all pass. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - `tests/runner-e2e/workflow-security.test.ts` has 5 failures that this change does not cause. They concern `pr-trusted.yml`, `chat-catalog-testops-case.yml` and `runner-full-stack-e2e.yml`. This PR does not touch those files. - I have not run a live controlled campaign on this branch. Controlled mode runs only from the default branch. The current 41-case run uses master and the existing variable. ## Risks - The committed price table has no expiry. A provider price change needs a reviewed edit to the table. The spend cap still limits the maximum spend. - If the committed lockfile does not match the package manifests, the controller build fails before provider work. CI already requires a frozen lockfile. - The spend cap is now $400 by default. It was $339 in the last manual policy. The cap is a reservation limit, not expected spend. The expected spend for 41 cases is about $5–20. - Only ACPX Claude Sonnet 5 has a committed allocation, as in the old variable. A new controlled model needs a committed price entry and allocation. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
01dd6ab336 |
fix(recovery): show the saved-work recovery step and record why it failed (#19260)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sometimes a failed run left files in a sandbox. Then Retry first
runs system recovery to save those files before the task continues.
> - Recovery has steps: confirm that the previous attempt stopped, save
its files, then finish.
> - The task page did not show these steps. People saw "Queued" or
"Trying again in…" with a retry time that kept moving.
> - Each unclassified failure of the stop-confirmation step was recorded
only as `provider_unavailable`, so a repeated deferral could not be
diagnosed.
> - This pull request shows the current recovery step on the task. It
also records why each failed attempt failed, in closed labels.
> - The benefit is that people can see what Retry waits for, and
operators can see why recovery deferred.
## Linked Issues or Issue Description
Refs #19249
**What happened?**
A person clicked Retry on a task whose failed run had a retained
sandbox. For about 7 minutes the task showed only "Queued", and the
retry time moved forward twice. Recovery deferred twice in the
stop-confirmation step. Each deferral was recorded only as
`provider_unavailable` with an opaque fingerprint, so the cause could
not be found.
**Expected behavior**
The task shows the recovery step that it waits for, such as "Checking
that the previous attempt stopped. Trying again in 30s…". Each failed
attempt keeps a closed-label cause, for example `sandbox_not_running` or
`transport_timeout`, with an optional errno-style code and HTTP status.
**Steps to reproduce**
1. Get a remote task run to fail after its sandbox was placed, so that
recovery owns saved work.
2. Click Retry while the sandbox provider does not respond to the stop
check.
3. The task shows "Queued" or "Trying again in…". The recovery evidence
records only `provider_unavailable`.
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandPaperclip
|
5f61392cd6 |
Reconcile dormant Codex predecessors through ordinary recovery (#19132)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native Runner keeps provider and controller journals between turns. > - A process can stop after a completed turn while those journals remain valid. > - The next owner must reconcile that source before it starts new work. > - This pull request reuses ordinary recovery and authenticated run attachment. > - Existing journals and owner admission keep retained work under its source. > - The benefit is conversation recovery without another snapshot format or native mode. ## Linked Issues or Issue Description Refs #19057. Built on merged #19129. This is the Codex recovery transport foundation. Process selection and controller scheduling remain separate changes. On normal startup, `paperclip-runnerd` selects durable transport from its connection arguments. With `--connect-url`, the Runner dials the controller. With `--listen-address 0.0.0.0 --listen-path /api/runner/v1/connect/<run-id>` and optional `--listen-port` (default `43127`), the controller dials the Runner. Both use the same durable entry point for fresh launches and recovery. The bootstrap ticket authenticates startup, and `recoveryIntegrityVersion` advertises a recovery capability. ## What Changed - Observe the existing recovered command journal before provider restoration. A saved dormant Codex session with no uncertain effectful command must still be dormant after restore. Ordinary active same-run recovery keeps its existing behavior. - Preserve prepared providers during snapshots, drain and cleanup. New completed-predecessor admission excludes prepared state. Existing cleanup and ordinary continuation keep their established behavior. - Advertise `recoveryIntegrityVersion: 1`. Reject the removed `--source-reconcile` flag. No separate native source mode or command whitelist remains. - Derive controller admission from the authenticated `authorityRunId` and actual Runner identity. Reuse retain-only settlement through attachment, authentication and checkpoint persistence. Preserve dynamic ownership across home configuration and later warm reuse. - Require the selected execution driver's trusted `sourceRecoveryVersion: 1` prelaunch attestation for a stopped predecessor. Binary metadata alone cannot authorize that launch. Ordinary live adoption keeps its existing path. - Support fresh authorized preparation from `recoveryLaunch: { requirements }` without replacing the source's ordinary cache. Require an authenticated owner throughout that transport's lifetime. Preserve immutable recorded policy through target preparation and later warm attachment; compare explicit settings using the provider factory's effective defaults. - Complete interrupted attachment at the existing fresh-snapshot boundary: persist the checkpoint, recheck current ownership, complete the external recovery callback, then release a real predecessor dispatch hold. Keep callback/checkpoint failures retryable and cancellation cleanup retained. - Preserve completed-session event delivery after target attachment, exact-thread history projections, source registration identity, exact command replay and nullable usage redaction. - Recheck current ownership and the queued run when a slow notification consumer presents a permission request. Suppress stale presentation while preserving the committed event and pending request for source-owner settlement. ## Verification Current head `e90bb379178db75f78e440c92e108f0bbe01e073` merges master `499712c68b3c` and is fully verified. [PR CI](https://github.com/paperclipai/paperclip/actions/runs/42058217937), [Runner Windows](https://github.com/paperclipai/paperclip/actions/runs/42058217814), and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42058217365) all passed. [Standard Greptile review 18](https://github.com/paperclipai/paperclip/pull/19132#issuecomment-6645302809) scores this exact head 5/5 with no unresolved threads. The earlier Apex review 16 applies to `c09ebb4a2a6c`, as recorded below; no new Apex review was requested after the documentation or master-forward changes. The conflict integration preserves upstream external cold-state restoration before completed-handoff inspection, predecessor ownership and pending-command checks, and the lost-source early-read exclusion. Live adoption still does not restore a backup. All 43 focused process cases passed across cold restore (including denied/foreign state), staged handoff, ordinary adoption and source recovery. Runner `tsc --noEmit`, parent-relative diff checks and added-line secret/personal-data scan passed. [Read the rewritten guide](https://github.com/paperclipai/paperclip/blob/e90bb379178db75f78e440c92e108f0bbe01e073/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md#recovering-codex-between-turns): it explains dial and listener startup, then follows completed run A through ordinary recovery and authenticated attachment to run B. The guide was rewritten in documentation commit `75db893fb9ab` and is unchanged by the master-forward. Its listener CLI test and both goal-update process tests passed. Earlier feature head `c09ebb4a2a` includes a process regression for a goal notification arriving between the restored goal response and the final thread response. Before the fix the Runner admitted the provider. The new tests prove that an active goal refuses admission and reaps the provider, and that a paused goal is delivered normally. Both process tests and five adjacent ordinary recovery cases pass locally. GitHub CI passed on the first attempt, including the main, Windows, and Docker workflows. Standard review 15 and Apex review 16 each scored this commit 5/5, with zero unresolved threads. [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41991764330) · [Greptile review](https://github.com/paperclipai/paperclip/pull/19132#issuecomment-6645302809) - The final ownership-race checks pass 7 focused cases, including owner changes during checkpoint persistence, handler publication and saved receipt recovery. - Native `codex_provider ordinary_recovery`: 6 passed, including saved/live mismatch, pending and indeterminate same-run goals, settled idle restoration and prepared reads/cleanup without a provider restart. - Native `paperclip-runnerd tests::`: 9 passed, including metadata and legacy flag rejection. - Native `interrupted_result_delivery_reconciles_exact_receipts_without_reexecuting_operations`: passed. - Final focused compatibility run: 24 process cases passed, including all 19 retained-settlement rows with queued snapshots, both home variants of A-to-B-to-C, ordinary attachment/suspension and environment-prepared close. - Existing `recovers an active second warm business run after deleting its controller context assets`: passed. - Runner `tsc --noEmit`, `git diff --check`, and added-line secret and personal-data scans passed. - Real native processes with the maintained fake Codex provider prove abrupt A exit, ordinary recover A, same-process attach B, a second successful turn, ordinary warm attach C and a third successful turn on the same thread. Both isolated and preconfigured homes pass. The isolated case also exercises environment option copying. - Additional process cases cover cacheless preparation, same-PID controller restart adoption, revoked preparation, retained tool-input admission, direct attachment, ordinary suspension and unexpected live goals. - A prior revised-head CI run exposed 8 retained-cleanup failures caused by prepared auto-restoration. That new behavior and its journal-binding code are removed. All 19 affected settlement variants now pass locally. The same CI run also had one environment-prepared close failure (focused check passes) and two unchanged Cloud sign-in E2E heading timeouts. Original logs remain recorded. Full suites and platform checks run in CI. Previous head `16772ad3e1abfdae9abcc64f8221fa3644ecee58`: [PR workflow 41981127140](https://github.com/paperclipai/paperclip/actions/runs/41981127140), [Runner Windows 41981126926](https://github.com/paperclipai/paperclip/actions/runs/41981126926), and [Docker Runner 41981126975](https://github.com/paperclipai/paperclip/actions/runs/41981126975) passed. Greptile review 14 scores this exact head 5/5 with no unresolved findings. The PR remains draft. CI evidence: an unchanged `idle-completed-history` local HTTP readiness case failed in the initial server 11/12 job (127092367616) and its first targeted retry (127096952128). The isolated case and the full 75-case file passed locally. The same-head debug retry (127103567635) passed, and the full workflow is green. Original failures remain recorded. No fixture diagnostic, assertion change, or timeout change was pushed. Focused review-fix validation: - 57 process cases passed across recovery, attachment and all 19 retained-cleanup variants. - Final staged matrix: 5 cases, each exercising queued/prepared/activated/completed boundaries, across owned connect/listen, legacy no-owner connect/listen, and retained cancellation cleanup. - Four completion-failure cases pass: synchronous/asynchronous callback failure, checkpoint failure before any callback, and owner rebind during persistence before any callback. Fresh-snapshot retry succeeds without duplicate provider work. - Saved-policy positive and explicit-conflict negative pass, including an authenticated compatible target grant and B-to-C continuity with omitted live settings. - Factory/Core checks: 34 passed, including effective ACPX defaults, real policy changes, already-dispatching target work and predecessor admission. - Ordinary adoption compatibility checks: 11 passed. Cacheless owner construction/revocation and policy checks: 6 passed. - Deferred permission ordering/forwarding: 30 passed. The new regressions reproduce owner changes before queueing and before slow-consumer delivery; matching-owner presentation and later source-owner settlement remain intact. - Runner `tsc --noEmit`, `git diff --check`, and full-PR added-line secret/personal-data scan passed at the corrected head. New tests reproduced both review findings and premature completion on failed persistence/changed ownership before the fixes. Full suites remain assigned to CI. ## Risks - Fake-provider tests do not prove vendor restore inertness or rollout fsync. - The process owner must prove stopped ownership, no successor and strict dormant provider admission before a predecessor launch. This PR does not select or stop host tasks. - Ordinary suspended active-goal continuation remains distinct from completed-predecessor admission. - Live dormant checks use bounded goal and thread reads. Unsupported or conflicting responses fail closed. - Prepared provider state is outside new predecessor admission. Reads and cleanup retain their baseline behavior. The feature targets completed between-turn session-open or exited sessions. - No tenant changes or image publication are included. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with local code and command tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this does not duplicate planned work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked the related public issue or PR - [x] I have not referenced internal issues or instance links - [x] My branch name describes the change - [x] Focused local tests pass - [x] I have added tests and updated documentation - [x] I have documented risks - [x] All Paperclip CI gates are green at the revised head - [x] Greptile is 5/5 with no unresolved comments at the revised head - [x] I will address review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
499712c68b |
fix(runner): restore cold state before inspecting completed handoffs (#19254)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner saves provider sessions and controller state across turns. > - A completed warm handoff leaves a receipt in the controller journal. > - A later cold continuation must restore external Runner state before it can inspect that receipt. > - The transport read the missing physical file before it called the restore hook. > - This pull request prepares admitted external state before that inspection for completed handoffs. > - Live Runner adoption and pending transitions retain their existing ownership checks. The provider thread can continue after a cold restore. ## Linked Issues or Issue Description Related: #19225 and #19157. This fix changes the shared transport read order. It does not depend on the open source-reconciliation extension in #19157. **What happened?** A cold continuation after a completed warm handoff could fail with `runner_remote_state_unavailable`. The saved controller journal was valid. The physical Runner state was absent because its restore hook had not run yet. A later automatic reattachment could then fail an allocation identity check. **Expected behavior** Restore admitted external state before inspecting a completed handoff. Retain the original provider thread. Refuse a denied restore or foreign session without starting provider work. **Steps to reproduce** 1. Open a Codex session and complete a warm handoff to a second run. 2. Suspend the session and retain the controller journal and provider backup. 3. Start a third run with no physical Runner state until the external preparation hook restores it. 4. Read the retained thread. Before this fix, the read fails before preparation. ## What Changed - Prepare external Runner state before inspecting a completed warm-handoff receipt. - Keep live Runner adoption, pending attachments, and active warm transitions on the existing exact-state path. - Add a real Runner and controlled Codex regression for successful restore, denied restoration, and a foreign restored session. - Document the cold-continuation order in the execution contract. ## Verification - Red: the new real-process regression fails on unpatched source because resume reads the absent `runner-state.json` before preparation. - Green: all three new cases pass. The restored thread keeps its identity. The controlled provider starts one thread and resumes it once. No turn is replayed. - Adjacent safety suites: 60 cases passed across warm attachment, interrupted attachment recovery, suspension, and cold restore. The new foreign-session case also passes separately. - Runner TypeScript and generated protocol checks passed. - Full workspace typecheck and build passed. Runner TypeScript checks passed again after the live-adoption guard. The full local test command remains in progress; all required CI test jobs passed on the current head. - The first CI run found two live-adoption regressions: an already completed staged handoff must not restore a backup while its Runner is live. The guard now excludes live adoption. All six cases across cold restore and staged handoff pass locally. The original CI failure remains recorded in run 42047027346. - Greptile reviewed the corrected head `f33a041c8902c39ca52b995d53128fc23b0b5499` at 5/5. There are no review threads or findings. All current-head CI gates passed. - Live staging verification remains pending normal deployment. No pinned serving image or bulk migration is part of this PR. ## Risks The change calls an existing external preparation hook earlier for completed handoffs. That hook must remain idempotent and must enforce restore authority. Live adoption, pending attachments, and active transitions do not gain this earlier call. The negative regressions assert that a refused restore and a foreign session start no provider work and do not change the authority journal. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
8227259d86 |
feat(task-chat): show the agent's native plan as a task list above the composer (#19252)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. The runner turns each harness's native plan into a canonical `plan.updated` event. > - The task chat did not show these plans. A small "Step 2 / 5" pill showed the checklist only on hover, and in live runs it received no plan data. > - Live tests found three breaks. The UI flattened `plan.updated` into tool rows. Headless Claude Code hides its task tools from current models. Codex ships its `update_plan` tool disabled. > - This pull request fixes the three breaks and replaces the pill with a docked plan card above the composer. > - The benefit is that users see what the agent finished, what it does now, and what comes next. The view is the same for Claude, Codex, Grok and Copilot. ## Linked Issues or Issue Description **Subsystem affected** Task chat UI (turn status island), native runner ACPX launch environment for Claude, and native runner Codex app-server launch arguments. **Problem or motivation** Harnesses report native plans, and the runner records them as `plan.updated` events. Users do not see them. The live transcript builder converts `plan.updated` into generic tool rows, so the plan steps are lost. Claude Code in headless mode does not offer TaskCreate/TaskUpdate for current models, so Claude never makes a plan. Codex 0.160 has `update_plan` disabled by default, so Codex writes its checklist into a Paperclip document instead. **Proposed solution** Keep the full plan snapshot in the live transcript. Enable the Claude task tools with `CLAUDE_CODE_ENABLE_TODO_TOOLS=1` in the ACPX launch environment. Enable the Codex plan tool with `tools.update_plan.enabled=true`. Show the plan as a docked card with clear rules for when it shows, collapses and hides. **Alternatives considered** Show the plan only in the hover popover (the old behavior). Users did not find it. Write the plan into the issue Plan document. That mixes a per-turn checklist with the reviewed planning document. **Roadmap alignment** This makes existing native runner plan events visible. It adds no new protocol or data model. **Additional context** Grok and Copilot needed no harness change. Cursor could not start in a local worktree instance (an unrelated startup admission problem), so this PR does not verify Cursor. ## What Changed - `ui/src/components/transcript/native-run-events.ts`: keep `plan.updated` as `provider_activity` with family `plan` and the full payload, not as a tool row. - `ui/src/components/task-chat/TaskChatTurnStatusIsland.tsx`: replace the hover pill with a docked plan card. It shows done, current, upcoming and blocked steps, a progress bar and file-change totals. - Visibility rules: hidden with no steps and after the run ends; collapses when all steps are done; header only for one-step plans and while a question or approval is pending; window around the current step for plans over 7 steps; collapsed by default on mobile. - `ui/src/components/TaskChatThread.tsx`: pass `defaultCollapsed` (mobile) and `compact` (pending runtime request), and reset the card per run. - `ui/src/components/task-chat/transcript-adapter.ts`: keep the `failed` step status. - `packages/paperclip-runner/src/drivers/acpx/environment.ts`: set `CLAUDE_CODE_ENABLE_TODO_TOOLS=1` for Claude launches. A host value cannot turn it off. - `packages/paperclip-runner/src/drivers/codex/codex-security-config.ts`: add `-c tools.update_plan.enabled=true` before the policy overrides and the `app-server` subcommand. - `ui/storybook/stories/tasks/task-chat-agent-plan.stories.tsx`: new story for each state. It uses the production composer dock, composer and question takeover. - Tests for all of the above. `ui/src/index.css` replaces the unused popover size token with a list max-height token. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/transcript src/components/task-chat src/components/TaskChatThread.test.tsx src/adapters/paperclip-runner` (1345 tests pass). - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx/environment.test.ts src/drivers/codex` (pass). - UI and runner typecheck are clean. `pnpm check:token-gates` is clean. - Storybook: open "Tasks and conversations / … / Task chat / Agent plan". The "All States Gallery" story shows every state. "Live Run Simulation" animates a run. - Live test on a local worktree instance with real API keys: a multi-step task for each harness. The card showed and updated during the run for Claude (Opus 5.5), Codex (gpt-5.6-sol), Grok (grok-4.7) and Copilot (gpt-5.6-luna). For Claude the card collapsed when all steps were done and hid after the run ended. - Codex probe: `codex exec` with the pinned 0.160 binary reports no `update_plan` tool by default and reports the tool with `-c tools.update_plan.enabled=true`. ## Risks - Claude and Codex now get their plan tools. Agents can spend a few tokens on plan updates. The tools are standard harness features. - The Codex flag goes before the web-search and project-instruction policy overrides, so the existing policy tail check still holds. - Runs that finish fast show the card only briefly. This is the intended hide rule. - Cursor plan display is not verified. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, browser automation) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e37ab23dbc |
fix: hold an agent's new work while its provider usage limit is active (#19246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 **What happened?** A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. **What did you expect?** A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. **Steps to reproduce** 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
21640b6d51 |
feat(connections): gate the built-in Google Workspace connector behind an experimental toggle (#19247)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Apps catalog lets operators connect external tools, including Google services. > - Paperclip has a built-in Google Workspace connector, but Google has not yet verified Paperclip's Google app. > - The built-in connector also hides the Composio and Arcade Google apps, because native Google definitions claim those app names. > - Users therefore see an unverified connector and cannot find the working aggregator Google apps. > - This pull request puts the built-in connector behind a default-off experimental setting. When the setting is off, the aggregator Google apps appear. > - The benefit is that users get working Google connectors now, and the team can still test the built-in connector on demand. ## Linked Issues or Issue Description **Problem** Google has not verified Paperclip's Google app yet. The Apps page still offers the built-in Google Workspace connector. The built-in Google definitions also suppress the Composio and Arcade Google apps (Gmail, Google Calendar, Google Drive, Docs, Sheets, Slides, Chat), so users cannot find them. **Proposed solution** Add the `enableGoogleWorkspaceConnector` experimental setting. The default is off. When it is off, hide and block the built-in connector, and show the aggregator Google apps. When it is on, keep the current behavior. **Alternatives considered** Delete the built-in connector until verification is complete. We rejected this option because the team must continue to test the connector. **Additional context** Composio documents Gmail (63 tools), Google Calendar (50 tools), and Google Drive toolkits that use Composio-managed OAuth. These toolkits do not need Paperclip's Google app verification. ## What Changed - Added the default-off `enableGoogleWorkspaceConnector` experimental setting to the shared types, validator, feature catalog, server defaults, and the managed-key list for Paperclip Cloud staging. - Added `experimentalFeatureKey: "enableGoogleWorkspaceConnector"` to the `google-workspace` app definition. The existing generic connector gate then hides the connector in the gallery and blocks its setup, discovery, execution, and agent suggestions when the setting is off. - Added a "Google Workspace connector (Paperclip)" toggle card to **Settings → Experimental**. - Apps page (`Browse.tsx`): when the setting is off, built-in Google rows without saved accounts no longer claim aggregator app identities. The Composio and Arcade Google apps then show as aggregator rows. The legacy Google slug filter now applies only to native rows, so the Composio `gmail` row is not dropped. - Saved per-service Google connections (Gmail, Drive, and others) are not gated. They stay visible and usable. - Updated `doc/connections/GOOGLE-WORKSPACE.md`. - Tests: added tests for the hidden and visible Apps page states, the settings toggle, and the definition gate. Existing built-in Google Workspace tests now enable the setting. ## Verification - `cd packages/shared && npx vitest run` passes (1055 tests). - `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/tool-access-service.test.ts src/__tests__/tool-gateway-service.test.ts src/services/gmail-tool-governance.test.ts src/services/google-chat-tool-policy.test.ts src/__tests__/instance-settings-service.test.ts src/__tests__/health.test.ts src/__tests__/instance-settings-managed-overlay.test.ts` passes (864 tests). - `cd ui && npx vitest run src/pages/apps src/pages/InstanceExperimentalSettings.test.tsx src/features/connections` passes. - `tsc --noEmit` passes for `packages/shared`, `server`, and `ui`. The `ui` run shows errors only in the untouched `storybook/support/feature-map` files. - `pnpm check:token-gates` passes. - Manual check: open **Apps** with the setting off. Search for "Gmail". The Composio Gmail row appears, and the built-in Google Workspace card does not. Turn on the setting in **Settings → Experimental**. The built-in card appears again. ## Risks - Existing built-in `google-workspace` connections stop running while the setting is off, because the generic gate blocks execution. This is intentional, so that the unverified Google app is not used. Turn on the setting on instances that must keep these connections. - Saved per-service Google connections are not affected. - When the setting is on, the behavior does not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in Claude Code with tool use (shell, file edits, web fetch). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
26391707fb |
fix(runner): never settle an OpenCode reply truncated at the output limit (#19242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode runs turn the provider's last assistant text into
the task's visible reply
> - A model can loop until it reaches the output limit. The provider
then reports `finish: "length"`
> - The driver used that cut-off text as the final answer, and the run
succeeded with a broken reply
> - This pull request treats a response that stopped at the output limit
as incomplete
> - The benefit is that users get the accepted completion summary or one
corrected answer, never a truncated wall of text
## Linked Issues or Issue Description
**What happened?**
In a QA staging run, an OpenCode agent on
`openrouter/deepseek/deepseek-v4.1-flash-1017` called `paperclip_finish`
correctly. Then it produced about 32,000 output tokens of repeated text
(`E2E_QUIP_QUIP_...`) until the output limit stopped it. The driver
selected this truncated text as the settled final answer. The run
succeeded, and the task reply was wrong.
**Expected behavior**
A response that the provider stopped at the output limit is not a
complete answer. If an accepted completion report exists, its summary is
the reply. If not, the driver asks for one concise final answer. If that
also stops at the limit, the turn fails.
**Steps to reproduce**
1. Run a native OpenCode turn where the model calls `paperclip_finish`.
2. Let the next assistant message run to the output limit (OpenCode
reports `finish: "length"`).
3. Before this change, that message text is emitted as the final
`agentMessage`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
03abe1f1a1 |
fix(native): keep the transport cause on a closed notification stream failure (#19240)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs read provider events from a shared session notification stream (Codex, OpenCode and ACPX drivers use it) > - When that stream closes, the driver fails the session with `notification_transport_failed` > - The run error was only "Provider notification transport failed closed." The real cause was only in a diagnostic event > - In a staging soak, many runs on OpenRouter and Grok agents showed this error. Operators read it as a provider fault. The diagnostic events showed a host-side cause: a stopped sandbox plugin worker ended remote runner supervision > - This pull request keeps the redacted, bounded cause on the terminal failure message, and does not name the provider > - The benefit is that a failed run shows why it failed, so operators do not chase the wrong provider ## Linked Issues or Issue Description **What happened?** Native runs failed with `native_provider_terminal_failed` and the error "Provider notification transport failed closed." The run did not show a cause. All failures in one cluster happened inside one minute, on several agents and models (OpenRouter DeepSeek, Grok). Each run's `harness.diagnostic` event had the real cause, for example `runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "<plugin>" is stopped`, or `native_adopted_runner_identity_unverifiable: runner liveness could not be revalidated`. The sandbox duplex transport reported a `provider_exit` loss about 30 seconds before (the read-only process probe budget). **Expected behavior** The terminal run error keeps the redacted cause. It does not blame the model provider when the cause is host-side runner supervision. **Steps to reproduce** 1. Open a native Codex-protocol session and start a turn. 2. Make the notification stream throw (for example, a supervision error). 3. Read the `session.failed` and `turn.failed` messages. Before this change they are the bare generic text. ## What Changed - `codex-session-notifications.ts`: the `notification_transport_failed` terminal message is now `Notification transport failed closed: <cause>`. The cause is redacted first, then whitespace is collapsed and it is bounded to 1,000 characters. The `Error: ` prefix is removed. The code and `recoverable: false` do not change. - New regression in `codex-protocol-integrity.test.ts`: one `session.failed`, the cause and code are present, the secret is redacted, the length is bounded, and `turn.failed` has the same message. - `doc/execution-semantics.md`: an additive note that this failure can come from host-side runner supervision, and that the message keeps the redacted cause. ## Verification - Red: before the fix, the new test failed: `expected 'Provider notification transport faile…' to contain 'runner_process_termination_unconfirmed'`. - Green: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/ src/conformance/local-provider-fault-matrix.test.ts src/native-session-runtime.test.ts` (565 passed). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: no new errors. The existing `src/eval/workflow-harness.ts` errors come from an unbuilt workspace package. - Staging evidence was read with read-only GETs only (run, events). ## Risks - Low risk. Only the message text changes. The failure code, `recoverable: false`, and the safe-replacement rules do not change. The server matches on the code, not on the message. - This PR does not add a retry in the same turn. Runner termination is unconfirmed when supervision is lost. A retry or replacement could start a second executor. The remote process probe already retries read-only checks inside a 30-second budget. Recovery stays with the existing ownership and recovery checks. - The cause text can include a plugin id. It goes through the existing provider-diagnostic redaction. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
TonioandClaude Opus 5.5
|
9e9a628a57 |
feat(ui): turn Artifacts V1 on for every instance and hide its toggle (#19164)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Artifacts show the output of agent work on the Artifacts page, in the panel Artifacts tab, and in the task thread > - #17228 added the Artifacts V1 presentation behind an experimental setting that is off by default > - The presentation is now ready for all users, so an opt-in toggle is no longer necessary > - A new default alone does not reach all instances, because each experimental write persists every flag > - This pull request removes the toggle and turns V1 on for every instance > - The benefit is one artifact presentation for all users, with an off switch for hosting operators ## Linked Issues or Issue Description Refs #17228 Refs #17263 **Is your feature request related to a problem? Please describe.** Artifacts V1 shows only on instances where an admin turned on an experimental setting. Other users see the previous cards. **Describe the solution you would like** Show Artifacts V1 on all instances. Remove the toggle from Experimental settings. **Describe alternatives you have considered** A new default without a change to the hook. This does not work, because many instances have a stored false that nobody chose, and no toggle would remain to change it. ## What Changed - Removed the Artifacts V1 card from Instance Settings, Experimental. - Changed `enableArtifactsV1` to default on in the schema, the feature catalog, and the settings service. - Changed `resolveArtifactsV1Enabled` to ignore a stored false. It honours false only when the value comes from managed config. - Kept the flag and the previous cards, so a hosting operator can hold V1 off. - Kept the key in the Cloud staging list, so a managed false does not hold V1 off on staging. - Updated unit tests, the Artifacts V1 plan, and the directions-lab Today reference. - Moved the attachment and upload e2e specs and the runner download helper to the V1 rows. - Added V1 tests for a Markdown review: expansion, deep link, and open, raw and download actions. ## Verification - `cd ui && npx vitest run`: one test file failed before its fix. That file tests the previous cards, and it now mocks the hook to off. I reran it with the other changed UI tests, and they pass. I did not repeat the full UI run after the fix. - `npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/text-attachment-tabs.spec.ts tests/e2e/artifact-uploads.spec.ts tests/e2e/artifact-tab-arrival.spec.ts`: 10 tests pass on V1. - `tests/e2e/cloud-auth.spec.ts` failed once in CI on a task heading. It does not touch artifacts, and 10 local runs pass. - `cd server && npx vitest run src/__tests__/instance-settings`: 183 tests pass. - `cd packages/shared && npx vitest run src/feature-catalog.test.ts src/validators/instance.test.ts`: 35 tests pass. - `pnpm run check:token-gates` and the `ui` and `server` typechecks pass. I did not run the full repository check. - In the artifact directions lab, with no stored settings, the Today right panel renders the V1 rows. - I did not test this on a live instance. After merge, confirm on staging that the Experimental page has no Artifacts V1 card and that all three surfaces show V1. ## Risks - All users see V1 after the next release, not only staging users. To revert, revert this pull request. - Users cannot turn V1 off. Only a hosting operator can, through managed config, and not on Cloud staging. - An instance with a managed `enableArtifactsV1: false` stays on the previous cards. The cloud harness source has no reference to this key, but I cannot read the live fleet configuration. - V1 surfaces render before settings load. An operator-held instance can briefly show V1 before the previous cards. - Admins who use the API can still write the flag. The write has no effect on unmanaged instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
037c58bb22 |
fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox running after a turn. The host keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the agent's `idleTimeoutMs`) > - The Daytona provider also stops an idle sandbox after `autoStopInterval` minutes. The default is 15 minutes > - On staging, the warm idle window was 900,000 ms (15 minutes). The two windows were equal > - A follow-up turn at the end of the warm window resumed a sandbox that Daytona was stopping. Daytona refused the resume in about 4 seconds, and the run failed at setup > - #19195 added a bounded retry for that refusal. It did not remove the cause > - This pull request makes the two windows non-overlapping by construction. Daytona derives its auto-stop for a warm reusable sandbox from the same warm idle value that the host uses > - The benefit is that a follow-up inside the warm window always finds a running sandbox, and idle sandboxes still stop ## Linked Issues or Issue Description Refs #19195 (bounded retry for a refused resume; this change removes the common cause). **What happened?** On a staging soak, three follow-up runs failed at setup with a refused sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the previous turn released the sandbox. The warm idle timeout was 900,000 ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the code kept these two values apart. **Expected behavior** A reusable sandbox that keeps a warm runner must not auto-stop inside the warm idle window or at its edge. The provider backstop must fire after the host's own idle handling, with a margin. Idle sandboxes must still stop. **Steps to reproduce** 1. Configure a Daytona sandbox environment with `reuseLease` and a warm runner with `runnerIdleTimeoutMs: 900000` (or an agent with `lifecycleMode: warm`, `idleTimeoutMs: 900000`). 2. Run a turn. Do not set `autoStopInterval`. 3. Before this change: the sandbox is created with `autoStopInterval: 15`, equal to the warm window. A follow-up near minute 15 races the provider stop. **Paperclip version or commit** `c656b95b6b` (current `master`). ## What Changed - Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`. For a reusable lease that keeps a warm runner, the effective auto-stop is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when the environment itself selects `runnerLifecycleMode: warm`. An explicit `0` (disabled), a longer configured value, `inherit` with a per-turn agent, per-turn, and non-reusable sandboxes keep the configured value. - Daytona plugin: use the effective value at sandbox creation. On resume, raise a shorter auto-stop on an existing sandbox. This update is best effort and bounded to 5 seconds; the #19195 retry still covers a refusal. The stale-handle refresh threshold keeps the configured interval, so a cached handle is not trusted for longer before the provider confirms the new interval. The warm window is not written to the saved normalized config. - Host (`runner-environment-lifecycle.ts`): add `resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective warm idle window (environment `runnerIdleTimeoutMs`, else the agent `idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`. - Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`): pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider driver on acquisition and on remote recovery resume. The driver adds it to the provider worker config only. Lease identity, reuse matching, and lease metadata are unchanged. - Daytona manifest: describe the warm floor on `autoStopInterval`. - `doc/execution-semantics.md`: document the contract. ## Verification - Red (before the fix, same tests, with the floor disabled): 5 failures in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The staging case failed with `expected 900000 to be greater than or equal to 1200000`. The resume case failed because `setAutostopInterval` was not called. - Red (host): the new `environment-runtime.test.ts` case first failed because the provider worker config had no `runnerIdleTimeoutMs`. The service wrapper dropped the field. The fix forwards it. - Green: `npx vitest run --config vitest.config.ts` in `packages/plugins/sandbox-providers/daytona`: 563 passed. This includes a setter that never settles (resume completes after the 5-second bound) and `inherit` with no forwarded window (explicit `autoStopInterval: 2` stays 2). - Green: `npx vitest run src/services/runner-environment-lifecycle.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242 passed. This includes recovery resume forwarding the admitted warm window. - The regression tests assert the relationship `autoStop >= warm idle + margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s) and for an explicit `autoStopInterval: 15`. They are not constant checks. - `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in the Daytona package passes. ## Risks - A warm reusable Daytona sandbox now stays running up to the warm window plus 5 minutes after its last activity, instead of 15 minutes. With a 15-minute window this is 20 minutes. Cost is bounded and idle sandboxes still stop. - The resume-time `setAutostopInterval` call is best effort. If it fails, the old interval stays and the #19195 retry still applies. - Other providers are unchanged. They receive one extra host-owned config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner` runs, and they ignore unknown keys. - No lease identity change, so existing retained sandboxes are still reused. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c656b95b6b |
fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox after a turn, so a follow-up
turn resumes the same sandbox and its files
> - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes
by default). The warm native idle window is also 15 minutes
> - A follow-up that reaches the sandbox during that stop gets a
provider refusal. The host keeps the sandbox ("the lease was preserved
and no replacement was created"), which is correct
> - But setup then records a generic `setup_failed`. No retry is
scheduled, no cause is recorded, and the reply that woke the agent is
effectively dropped
> - This pull request gives that refusal its own failure code and routes
it through the existing bounded bootstrap retry. The retry resumes the
same recorded sandbox
> - The benefit is that a follow-up continues a few seconds later on the
same saved sandbox, instead of stopping the task with no next action
## Linked Issues or Issue Description
Refs #18852 (the bounded bootstrap retry that this change reuses).
Related: #19152 and #19187 (copy-back causes), #19178 (Daytona
errored-sandbox recovery before stop-only cleanup), #19182 and #19180
(recovery holds). This change does not overlap with them: they cover
cleanup, copy-back, and recovery after a run. This change covers a
resume refusal before provider work starts.
**What happened?**
On a staging soak, two follow-up turns failed at setup after a
successful first turn:
- A board reply woke the agent. The run failed after 4.9 seconds with
`setup_failed`: `Reusable sandbox lease "…" could not be resumed; the
lease was preserved and no replacement was created.` The issue stayed in
review with no recovery action. The reply was not delivered.
- An earlier run on another task failed with the same message.
Both resume attempts started 15 minutes 8–25 seconds after the previous
turn released the sandbox. The first turns used `keep_running` with a
warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval`
is also 15 minutes. So the resume reached the sandbox while the provider
was stopping it. The provider refused within seconds; this was not a
long resume timeout. The provider cause was not recorded, so this timing
is the strongest evidence. The exact provider state remains an
inference.
**Expected behavior**
A refused resume before provider dispatch is a temporary infrastructure
condition. The host retries it with a small, bounded budget. Each retry
resumes the same recorded sandbox. A refusal does not prove that the
sandbox is lost, so the host never destroys it or creates a replacement.
Only an explicit provider not-found or a workspace identity mismatch
with a verified backup permits replacement (unchanged). The run records
why the provider refused, without provider text.
**Steps to reproduce**
1. Run a warm remote turn that keeps its sandbox.
2. Make the provider's `environmentResumeLease` throw a non-transient
error (for example, a 400/409 while the sandbox stops).
3. Wake the agent with a reply.
4. Before this change: the run fails with `setup_failed`, no retry is
scheduled, and the error has no cause.
**Paperclip version or commit**
Seen on staging at `5b99bdae`. The touched code is unchanged on current
`master` (`f7a4c5678f`).
## What Changed
- Move `ReusableSandboxResumeError` to
`server/src/services/reusable-sandbox-resume-error.ts` (re-exported from
`environment-runtime.ts`). Give it the code
`reusable_sandbox_resume_failed`.
- Add the provider refusal to the error message by closed labels only:
`(cause=<label> status=<http> code=<errno>)`. The labels come from the
existing transport classifier, now exported as
`sandboxProviderFailureLabel`. Provider text, paths, and IDs are never
copied.
- Carry the provider status across the plugin worker RPC. A JSON-RPC
error has only a message and a numeric RPC code. The worker host now
attaches a validated `paperclip/environment-resume-error/v1` envelope
(HTTP status and an allowlisted errno only) to `environmentResumeLease`
failures. The host re-validates it and adds it to the recorded cause.
- In setup-failure settlement, find this error through the orchestrator
wrapper. Record `reusable_sandbox_resume_failed` only when provider
dispatch did not start and the run has no native session or runner
identity (the same guard as `remote_workspace_prepare_timeout`).
- Add the code to `isTransientWorkspacePreparationCode`. The existing
scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop,
pause, and dependency checks.
- Exclude the code from generic continuation recovery and from legacy
reconciliation (no unknown provider actions; bootstrap evidence proves
none started), like the template.
- Add a specific stranded-recovery notice: "Saved sandbox could not be
resumed".
- Document the contract in `doc/execution-semantics.md`.
## Verification
- Red (before the fix, same tests): 6 failures. The `fresh` case
recorded `setup_failed` and did not schedule a retry. The error had no
cause. Legacy reconciliation held the run. Generic recovery granted
another budget.
- Green: `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery/
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/environment-runtime.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass.
Also `src/services/recovery/stranded-notice` and
`src/services/remote-workspace*`: 411 pass.
- Safety regressions:
- `run-completion.test.ts`: only a pre-dispatch legacy failure with no
native identity gets the code and a retry. `started`, `retained`
(native), and an untyped copy of the message stay `setup_failed` with no
retry. The recorded error contains the closed cause and not the provider
path.
- `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent
schedulers across a restart create exactly one successor per attempt.
The budget ends after two attempts. Stop, pause, and a new dependency
cancel the scheduled retry. Generic stranded recovery creates no extra
run after exhaustion.
- `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through
the real worker RPC, the `JsonRpcCallError` has no status, and the
envelope gives 409 and the errno. Provider text does not cross. Other
methods get no envelope. Malformed envelopes are rejected.
- `environment-runtime.test.ts`: a throwing resume still preserves the
exact lease. The runtime does not call destroy or acquire. The resume
throws a `JsonRpcCallError` whose message has no status. The thrown
message names `cause=provider_error status=409` from the envelope, and
not the provider text.
- `node scripts/typecheck-low-memory.mjs` passes. `git diff --check`
passes.
- Not verified: a live staging run at this revision. The original tasks
were not retried. This PR is not deployed.
## Risks
- Low. The change affects only runs that fail with this typed error
before provider dispatch. Each retry resumes the same lease and inherits
the source wake context, so a reply is delivered once and no second
executor starts.
- If the provider cannot start the sandbox for longer than about 90
seconds, the two retries end and the task gets the specific recovery
notice. The sandbox and its saved files are kept.
- The error message now has a parenthetical cause before the semicolon.
No code parses this message.
- An issue in review does not get the immediate stranded notice
(unchanged existing behavior). The retry, not the notice, delivers the
reply in that case.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads of staging diagnostics).
Extended reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandPaperclip
|
895cee28b3 |
fix(daytona): recover an errored sandbox before stop-only cleanup (#19178)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs use sandbox provider plugins. Daytona is one of
them.
> - After a failed run, Paperclip stops the sandbox and keeps its files
("stop-only cleanup"). A task can continue only when every remote lease
of the failed run has a provider stop receipt.
> - Daytona rejects Stop for a sandbox in its `error` state. A sandbox
that filled its disk went to that state. The plugin called Stop on it,
so every cleanup sweep failed with the same generic error.
> - The lease stayed in pending cleanup without a stop receipt. Each
Retry or comment waited forever with reason `remote_cleanup`. A manual
Daytona Recover was the only fix.
> - This pull request makes the Daytona stop-only hook recover a
recoverable sandbox once, with a bound, and then stop it. When it cannot
stop the sandbox, it returns a typed failure. The server escalates a
terminal failure at once instead of retrying it like a transient error.
> - The benefit is that a full-disk sandbox no longer blocks its task,
and a sandbox that cannot recover is visible to operators on the first
attempt.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A task sandbox filled its 10 GB disk. Daytona then reported the sandbox
as `state: error`, `desiredState: started` and `recoverable: true`. Its
`errorReason` was a Docker `no space left on device` error. The Daytona
`stopLease` hook calls Stop on any sandbox that is not `stopped` or
`archived`. Daytona rejects Stop in the `error` state. The pending
cleanup sweep ran about every 30 minutes, and it failed on every pass.
The lease stayed `pending_cleanup` with `cleanup_status=failed` and no
stop receipt. Continuation admission
(`server/src/services/explicit-native-continuation.ts`) requires a stop
receipt on every remote lease of the failed run. So each Retry and
comment was deferred with "Waiting for the previous environment to stop.
Your message will start automatically." It never started.
A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The
sandboxes reached `started` in about one minute. The next sweep stopped
them, recorded the receipt, and the tasks continued.
**Expected behavior**
Stop-only cleanup recovers a recoverable errored sandbox and then stops
it. When the sandbox cannot be recovered, cleanup returns a clear, typed
failure that the server can tell apart from a transient error. Cleanup
never deletes the retained sandbox, and it never loops on Recover.
**Steps to reproduce**
1. Run a task in a Daytona sandbox until the sandbox fills its disk and
Daytona shows `state: error`, `recoverable: true`.
2. Let the run fail, so the server records a `sandboxStopAndRetain`
intent and calls `environmentStopLease`.
3. Before this change, Stop fails with `Sandbox failed to stop with
status: error` on every sweep, and the task cannot continue.
4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox
whose Stop fails from `error`.
**Paperclip version or commit**
Observed on a hosted deployment. Reproduced in unit tests on `master` at
`6452e964a36`.
**Deployment mode**
Managed Daytona sandbox execution with plugin-backed stop-only cleanup.
Related: #18172 (Daytona exports under disk pressure) and #19050
(session reset on every Daytona restart path). This PR does not change
the resume path in `ensureSandboxStarted`, which already recovers a
recoverable sandbox.
## What Changed
- **Daytona plugin
(`packages/plugins/sandbox-providers/daytona/src/plugin.ts`)**: after
`stopLease` confirms the retention policy, a sandbox in `error` goes
through `recoverSandboxForStop`.
- Recoverable: the hook calls `sandbox.recover()` once. The SDK waits
for `started`. The plugin bounds the wait to 20 s, plus the existing
liveness margin for a hung connection. Then the hook stops the sandbox
normally and returns the usual `stopped` receipt.
- Not recoverable: the hook makes no Recover or Stop call. It throws
`PluginEnvironmentStopError("resource_unrecoverable")`.
- Recover fails: the hook reads the provider state once. A sandbox back
in `error` gives `recovery_failed`, or `resource_unrecoverable` if
Daytona now marks it so. A sandbox that is still starting, or a failed
read, gives `recovery_incomplete`.
- The recovered sandbox fails again before Stop finishes (for example,
the disk fills again): a fresh read that shows `error` gives
`recovery_failed`. The hook never calls Recover a second time in the
same call.
- The hook never deletes or archives the sandbox. The non-error stop
path is unchanged.
- **Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`,
new)**: `PluginEnvironmentStopError`, `environmentStopErrorData` and
`readEnvironmentStopFailure`. This follows the existing
`PluginEnvironmentRunnerIngressError` pattern. The envelope holds only
one closed reason. The reader validates the reason and derives
`terminal` from it. `resource_unrecoverable` and `recovery_failed` are
terminal. `recovery_incomplete` is not.
- **Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`)**:
forwards that envelope as JSON-RPC error `data` for
`environmentStopLease` only.
- **Pending cleanup sweep
(`server/src/services/heartbeat/recovery.ts`)**: the catch reads the
validated envelope.
- On a terminal failure, it escalates on the first attempt. It writes
the one-time "needs operator attention" warning and uses the existing 30
minute retry interval. Before, this happened only at the fifth attempt.
- The failure log uses `errorKind: "provider_stop_terminal"` and
includes the closed `providerStopFailure` reason. The sweep still never
logs exception fields.
- The lease stays `pending_cleanup`. Automatic cleanup continues, and an
explicit Retry still bypasses the cooldown. Untyped and transient
failures keep their current behavior.
- **Docs**: the SDK README ("Stop failures"), the Daytona README,
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and
`doc/native-workspace-finalization-recovery.md` describe the new
behavior.
Shared contract note (module boundaries): the SDK envelope is the
smallest addition that lets the host tell a terminal provider stop
failure from a transient one. The worker host drops all other error data
across the RPC boundary, so the existing contracts cannot carry the
signal. The only consumer is the pending cleanup sweep. When a provider
does not throw the new error, or an older host ignores it, behavior is
unchanged. This PR does not touch
`server/src/services/workspace-recovery.ts` or
`sandbox-stop-and-retain.ts`.
## Verification
All commands ran in a clean worktree from `master`. Vitest ran from the
real path.
- Red on `master` (only the new tests added):
- Daytona, with `plugin.ts` from `master`: `npx vitest run --config
vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1.
All 7 new tests fail. The main case fails with `Sandbox failed to stop
with status: error`, which is the production error.
- SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run
tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the
typed case fails).
- Server, with `recovery.ts` from `master`: `npx vitest run
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal
provider stop failure"` exits 1. All 3 cases fail (`errorKind:
"destroy_failed"`, no escalation).
- Green on this branch:
- `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config
vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm
typecheck` exits 0. `pnpm build` exits 0.
- `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm
--filter @paperclipai/plugin-sdk typecheck` exits 0.
- `server`: 15 related suites exit 0 (1498 passed): pending cleanup
sweep, `heartbeat/recovery`, environment runtime, environment cleanup
work, native startup cleanup, native remote failure retention, observed
workspace preservation, retained native session cleanup, idle sleep
safety, explicit native continuation, task retry, workspace recovery,
native failed and cancelled workspace preservation, and Daytona close in
the plugin worker manager.
- `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm
check:module-boundaries` exits 0.
- New tests:
- Daytona: a recoverable errored sandbox is recovered once
(`recover(20)`) after the retention check and before Stop, then stopped.
An unrecoverable one returns `resource_unrecoverable` with no Recover,
Stop, start, delete or archive call. A failed recovery makes exactly one
Recover call for each outcome (`recovery_failed`,
`resource_unrecoverable`, `recovery_incomplete`). A Recover call that
never settles is cut off by the bound. A recovered sandbox that fails
again before Stop gives `recovery_failed` with no second Recover.
- SDK: the envelope holds only the reason. A forged or non-SDK error
gives no data. Unknown reasons, accessor fields and a sender-supplied
`terminal` flag are not trusted. The worker host forwards the typed
failure and drops a plain error's detail.
- Server: terminal reasons escalate on attempt 1 with the long retry and
one operator warning. `recovery_incomplete` keeps the short retry. The
lease stays `pending_cleanup`, and the provider message never reaches
the logs.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`
and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live
Daytona probe ran. The mocks model the Daytona SDK `recover()` contract:
it waits for `started` and rejects when the sandbox returns to `error`.
## Risks
- Time budget: the host gives `environmentStopLease` at most 60 s. A
recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus
short reads. A slower recovery returns `recovery_incomplete`. The SDK
call can continue at Daytona after that, so the next sweep finds the
sandbox started and stops it normally. If the host deadline fires while
the worker still stops the sandbox, the next sweep sees `stopped` and
records the receipt.
- Classification: after a failed Recover, the plugin cannot tell a
rejected request from a sandbox that started and failed again. Both give
`recovery_failed` (terminal) when Daytona still shows `error`. A hung
Recover call with a sandbox still in `error` is also terminal. Terminal
failures still retry every 30 minutes, and each retry makes at most one
Recover call. An explicit Retry also makes one attempt. There is no
tight loop. A sandbox that fails Recover every time gets about two
Recover calls an hour until an operator acts.
- Recover restarts the sandbox container before Stop. The plugin runs no
command in it, and the retention policy is confirmed before Recover.
- The continuation message for a terminally failed lease still says the
message "will start automatically". This PR does not change admission. A
follow-up can use the typed reason there.
- Compatibility: a host without this change ignores the new error data
and keeps the old retry behavior. A plugin without this change sends no
typed data.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, code execution for tests and typechecks).
The runtime does not expose the context window size to the agent.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandPaperclip
|
85e4d12856 |
fix(evals): report controlled protocol cases from safe records (#19070)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner protocol qualification records real provider behavior in TestOps. > - Controlled attempts keep raw records and accounting in private storage. > - A live case passed, but a later public-render command rejected its controlled context. > - The missing public report caused the final result to be stored as broken. > - This pull request creates the closed TestOps result from existing sanitized records inside the private report job. > - The benefit is correct result reporting while the privacy guard stays effective. ## Linked Issues or Issue Description Refs #18985 and #19011. The related-PR search found no separate repair for this failure. **What happened?** [The live canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378) passed all four provider checks. Its controller verified cleanup and closed its request accounting. The private canonical report recorded one passed case. The next command used `--public-viewer` while the controlled manifest remained set. The eval program rejected it with `controlled campaign evidence must remain private`. The closed exporter then had no public payload and correctly stored a broken result. **Expected behavior** Report the original case grade from sanitized structured records. Keep raw controlled records private. Keep the original checks, source bindings, and missing-evidence rules. **Steps to reproduce** Run one controlled protocol case with TestOps reporting enabled. Let the private canonical report finish. The following public-render step fails before the closed result can read the grade. **Paperclip version or commit** Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`. **Deployment mode** GitHub-hosted Linux, controlled ACPX Claude protocol qualification, first workflow attempt, zero automatic retries. ## What Changed - Add a narrow `export-runs` path to the existing TestOps helper. Bind the sanitized score and artifact to the exact attempt, case, model, provider, driver, and public-projection schema. Reuse the existing strict grade and check validation. - Keep sanitization inside the private report job. Use structured records for controlled TestOps reporting. Keep the public viewer for legacy publication. - Add record identity, privacy, grade, command, and workflow routing controls. Wire the existing helper and workflow-security files into the required Node registry. Document the reporting path. ## Verification - 43 focused campaign/sanitizer, TestOps and workflow security tests passed. The command regression writes a raw attempt, calls the production sanitizer, and runs `export-runs` on its output. It also repeats that chain after a failed-check mutation. Both new-control files now run through required `test:typescript:prep`, not only the optional publisher script. - The actual required prep ran locally with provider credentials excluded. The eval-kernel and Rust builds passed. Its Node registry executed the new producer-to-export and security controls by name: 403 passed, 2 failed, 11 existing skips. The two original package-contract failures cannot resolve this isolated clone's Codex ACP and Claude ACP package links. This is not a green local full-prep claim. The first attempt also stopped at a missing Node-types link, before an ignored clone-local link to the locked cached version restored that prerequisite. - 72 release workflow tests passed with the existing locked pnpm 9.15.4 wrapper and local loopback permission. The first sandbox run failed because loopback fixtures could not listen; the default local tool path also failed two install fixtures. No test assertions were changed to resolve these failures. - The actual repaired workflow shell ran locally against the retained canary campaign and sanitized records. It emitted one closed passed result with all four original checks. No provider request or upload occurred. - Missing or failed checks still reject a pass. Private-only markers stay absent from the closed output. Wrong model, provider, driver, attempt, case, and schemas reject. - The unmodified, pinned eval program still rejects a controlled public-viewer command. - Actionlint and `git diff --check` passed. [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886) passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates, all four Runner lanes, typecheck, build, backend and browser shards passed. Package suite: 4951 passed, 10 existing skips across 297 passed files and one existing file skip. - The required static log contains passing lines for both the production sanitizer-to-export command regression and the closed-projection security control. The prior optional-script registration did not run those tests in required CI; the one script-entry fix now enforces them. Provider/runtime and workflow bytes are unchanged by the verification-only follow-ups. - Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues. The separate Paperclip Review requested a producer-to-consumer regression. That regression and its required-CI wiring are complete. The addressed thread is resolved; no open review threads remain. - No fresh deployed reporting pass is claimed. The failed workflow and its stored broken result remain unchanged. ## Risks The exporter reads structured records inside the private job. It copies only the closed result fields and requires the existing sanitized marker and identity checks. The final reporting job still receives only those files and verifies the saved result bindings. The Python privacy guard, provider path, case assertions, budgets, deadlines, and retries are unchanged. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The inherited runtime reports a 475,000-token context window. Capabilities used: reasoning, code changes, and repository, shell, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5fdfd3f17a |
fix(recovery): bound repeated saved-record response bodies (#19168)
## Thinking Path > - Paperclip manages agent tasks and their saved execution history. > - A continuation carries completed API receipts to prevent duplicate work. > - Repeated saves can copy large record bodies into each new provider prompt. > - The resulting command can exceed the durable PRP frame limit before useful work starts. > - This change projects large successful saved-record responses as compact references. > - The original receipts and authenticated user direction remain available. ## Linked Issues or Issue Description **What happened?** A task with many saved document and comment responses can fail with `command_payload_too_large`. Same-run recovery can then show a runner identity failure that hides this first cause. Completed response bodies repeat content already stored in task records. **Expected behavior** Continue from saved records with completed-action evidence that fits the runner command. Keep user messages, answers and approvals intact. Keep the full original receipts. **Steps to reproduce** 1. Save 39 successful document, comment and task-write responses in a run's API receipt ledger. 2. Give each response a large record body. 3. Build and render a continuation, then queue it through the real durable PRP journal. 4. Before this fix, the regression throws `command_payload_too_large` with a 4.2 MB prompt. Related work: #16980 pages oversized assigned tool results. This change addresses earlier response bodies repeated in a later continuation. It does not change tool-result paging or protocol limits. ## What Changed - Project response data over 4 KiB for three known successful record-write operations. Keep record, revision and status metadata, HTTP outcome, receipt identity and a data digest. - Keep small, failed, unknown, mismatched and foreign-company results in their existing form. - Explain the reference format in shared prompt guidance and the execution contract. - Add database and real-journal regression coverage. Verify the original receipt ledger remains unchanged. The continuation service owns the projection. The shared adapter helper only explains the reference to native and legacy prompt consumers. There are no new dependencies, provider calls, timers or background scans. ## Verification - The unfixed database regression fails at the real PRP command limit with `command_payload_too_large`. - Continuation builder, native execution input and native continuation suites: 118 tests pass. - Shared prompt consumers: 142 tests pass. - Module boundaries and diff whitespace checks pass. - Greptile: 5/5 on `8109e7361c5b5e38f9aad35af9bf4ad7bdc5f338`, with no unresolved review threads. All CI checks pass. - Full workspace `pnpm -r typecheck` and `pnpm build`: passed. - Hosted verification: pending deployment. The source fix does not itself resume the affected hosted task. Local setup failures remain accounted for: the fresh checkout lacked an ignored root SDK link and embedded PostgreSQL library symlinks after an offline install that skipped lifecycle scripts. The maintained workspace-link helper, SDK build and platform hydration script repaired those prerequisites. A later test expectation assumed JSON insertion order across PostgreSQL JSONB; it now checks the digest against the unchanged stored receipt. These failures did not change runtime limits or recovery guards. ## Risks Large responses from these three successful operations now use explicit saved-record references in continuation context. Agents must read the record or source receipt when full content is needed. This is not a universal prompt-size limit: user history, unknown operations and other inputs retain their existing representation. No migration, permission, credential, provider-stop or replay policy changes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, through Codex in T3 Code. Used for investigation, implementation and local verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the targeted tests locally and they pass - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address review comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
7f9ef38273 |
fix(runner): report the upstream cause of OpenCode provider failures (#19166)
OpenCode reports some OpenRouter failures as an UnknownError whose message is the provider's JSON error body. The provider failure now reads its HTTP status, error type and message from that body, so the run shows the upstream message instead of raw JSON. The OpenCode nativeTurnFailure diagnostic now carries the provider failure's category, retryability and HTTP status (closed values only; no provider text or code), and recognizes the driver's opencode_empty_response code. Run status, error codes and retries do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
737db73275 | fix(native): measure resumed run usage from the prior run's total (#19161) | ||
|
Devin FoleyandClaude Opus 5.5
|
6452e964a3 |
fix(runner): ignore replayed history for restored terminal turns (#19035)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner executes native provider sessions (Codex app-server
and ACPX) and can resume a run from an exact persisted checkpoint on a
replacement sandbox.
> - When a turn completes without a semantic result, the server
schedules one same-run disposition recovery from that checkpoint.
> - On the replacement sandbox, the restored runner can deliver the
committed history of the checkpointed turn again as live notifications,
before the disposition turn starts.
> - Between turns there is no active turn. The identity classifier only
marks settled-turn notifications as `stale_turn` while a different turn
is active, so the replayed `turn/started` reaches the live-turn guard
and fails the session with `turn_binding_mismatch` ("Provider started an
unexpected turn."). Recovery then stops with
`native_event_replay_conflict`.
> - This pull request makes the driver drop replayed history for turns
that were already terminal in the restored checkpoint.
> - The benefit is that exact-checkpoint disposition recovery can
continue instead of failing closed on its own history.
## Linked Issues or Issue Description
**What happened?**
A hosted ACPX run (Claude model) completed its first turn without a
semantic result. The server persisted an exact-checkpoint disposition
recovery. The second attempt restored the harness state from the
verified failover backup on a new sandbox. About 3.7 seconds after the
runner transport was ready, and before any `turn.submitted` event, the
driver failed with `turn_binding_mismatch: Provider started an
unexpected turn.` The run then stopped with
`native_event_replay_conflict` and required workspace repair.
**Expected behavior**
Notifications that only replay a turn that the checkpoint already
recorded as terminal must not fail the resumed session. The disposition
turn must start normally.
**Steps to reproduce**
1. Open a session, start turn `turn-1`, and complete it without a
semantic result.
2. Take the snapshot and recover the session on a new transport.
3. Deliver `turn/started`, `item/*` and `turn/completed` notifications
for `turn-1` again, before or during the disposition `turn/start`.
4. Before this change, the session fails with `turn_binding_mismatch`.
If the replay arrives while `turn/start` is pending, the driver binds
`turn-1` as the new turn and fails with "Codex turn identity changed
during start".
**Paperclip version or commit**
Observed on a staging release built from `c2163b3ecf08`. Reproduced on
`master` at `89703c3`.
## What Changed
- `CodexSessionState` records the turn IDs that were terminal in the
restored checkpoint (`restoredTerminalTurns`). It also adds
`isRestoredTerminalReplay(turnId)`. This is true only when the turn was
restored as terminal, is not the active turn, and the session has not
failed.
- `codex-session-notifications.ts` drops `turn/started`, `item/started`,
`item/completed` and item or turn delta notifications for such turns. It
emits no turn or item events for them. It emits a bounded
`provider_notification_identity` diagnostic with `classification:
"restored_terminal_replay"`, and shares the existing limit of 32
diagnostics.
- A replayed `turn/completed` is unchanged. It still goes through
`mapTerminalTurn`, which compares the terminal fingerprint and fails on
a conflict.
- New regression file `codex-app-server-driver.terminal-replay.test.ts`.
It replays history before and during the disposition submission. It also
checks that a live session still fails closed on `turn/started` for a
turn that ended in that same session.
## Verification
- Red before the fix, on `master` with only the test file added:
- `npx vitest run
src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts` gives
2 failed and 1 passed. The "before" case fails with
`NativeProviderTerminalFailure: Provider started an unexpected turn.`
(`providerCode: turn_binding_mismatch`). This is the production error.
The "during" case fails with `Codex turn identity changed during start`.
- Green after the fix: the same command gives 3 passed.
- Related suites: `npx vitest run src/drivers/codex
src/drivers/acpx/codex-acpx-driver.test.ts
src/conformance/harness-driver.test.ts
src/backends/harness-driver-backend.test.ts
src/live/live-session.test.ts` gives 567 passed and 1 failed. The
failure is `codex-boundaries.test.ts > defers provider-owned workspace
existence without weakening its assignment`. It also fails on unchanged
`master` in my sandbox, because the sandbox workspace path triggers the
sensitive-path check. It is not related to this change.
- `tsc -p tsconfig.json --noEmit` shows no errors in the changed files.
The only errors come from the unbuilt
`@paperclipai/paperclip-eval-kernel` package in
`src/eval/workflow-harness.ts`, because I did not run the eval build
prep step locally.
## Risks
- The change relaxes the guard only for turn IDs restored as terminal
from the checkpoint. Notifications for those IDs are now ignored and do
not fail the session. A provider cannot use a settled turn ID for new
work, so these notifications cannot be new provider activity.
- Live post-terminal notifications, pre-turn notifications, foreign turn
IDs, `paperclip/runResult`, and tool calls keep their current
fail-closed behavior. The new test covers the live post-terminal case.
- Limitation: this change makes the driver tolerant of the replay. It
does not change why the runnerd transport cursor replays the committed
window on exact-authority recovery. The transport-level reason was not
reproduced here. The new diagnostic makes the event visible in run logs,
so later evidence can confirm or reject that path.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as an agent with
tool use, shell and test execution, and repository search.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandClaude Opus 5.5
|
6fa54d5577 |
fix(codex-local): route typed ACP credential rejections to sign-in recovery (#19146)
Classify a typed Codex ACP `access` failure as `acpx_auth_required` when the provider text reports `401 Unauthorized`, `invalid_api_key`, or an incorrect/invalid API key. Other `access` failures keep `acpx_turn_failed`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9e5039f6c8 |
fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description **What happened?** A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. **Expected behavior** A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. **Steps to reproduce** 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. **Paperclip version or commit** The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. **Deployment mode** Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3a72efebe5 |
fix(native): confirm no runner process when remote staging fails before launch (#19059)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native sandbox runs supervise a remote `paperclip-runnerd` process. When that process exits during a turn, the transport restarts it inside a reconnect grace window. > - The server launcher stages the runtime assets first. Then it sends one launch RPC that starts runnerd. > - When staging failed, the launcher reported the exit as unconfirmed. This happened although no launch RPC was sent and no process could exist. > - Reconnect recovery treats an unconfirmed exit as a possible second owner. So it stopped at once with `runner_process_termination_unconfirmed` and did not retry. > - This pull request confirms the exit for failures before the launch RPC. Failures after dispatch stay unconfirmed. > - The benefit is that a transient staging failure, such as a full sandbox disk, uses the normal bounded retry. A real failure ends with `transport_reconnect_grace_exceeded` and keeps the staging error as the detail. ## Linked Issues or Issue Description **What happened?** On a hosted staging deployment, 7 native ACPX runs failed in 24 hours with `Provider notification transport failed closed.` (`native_provider_terminal_failed`). In every run, the runner process exited during the turn. The replacement launch then failed while it uploaded the runtime context into the sandbox: `runner_process_termination_unconfirmed: replacement runner health could not be verified: ... .paperclip-upload-<uuid>.tar.gz: create: ... no space left on device`. The sandbox disk was full. No replacement process was started. But the launcher said that it could not confirm the exit. So recovery did not retry inside its grace window. **Expected behavior** A staging failure before the launch RPC proves that no replacement process exists. Recovery must retry within the configured reconnect grace. If the failure continues, the run must end with the reconnect-grace failure and the staging error. **Steps to reproduce** 1. Create a remote runner launcher whose `ensureArtifact` step throws, for example with an ENOSPC upload error. 2. Wait for the handle completion to reject. 3. Before this change, `isExitConfirmed()` returns `false`, and no launch RPC was sent. The transport then fails closed with `runner_process_termination_unconfirmed`. **Paperclip version or commit** Observed on staging revision `89703c3`. Reproduced on `master` at `be20d4ad97`. Related: #19035 and #19014 change other runner paths (replayed terminal turns and redacted external tool results). They do not change remote launcher exit evidence. #18999 adds a free-space reserve for host-volume seed copies. It does not change sandbox disk use or this classification. ## What Changed - `createRemoteRunnerProcessLauncher` records when it dispatches the launch RPC. If the completion fails before that point, the handle reports `isExitConfirmed() === true`. The original error is kept unchanged. - A failed or timed-out launch RPC still reports an unconfirmed exit, because the RPC may have started runnerd. - Added a launcher regression for a staging failure before dispatch. - Added safety cases for a failed and a timed-out launch RPC. - Added a transport regression with the real `paperclip-runnerd`. A replacement whose staging fails is retried, and the next launch restores the durable session without a notification failure. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote runner process supervision"` failed `confirms no replacement process exists when staging fails before the launch RPC` with `expected false to be true`. The two launch-RPC safety cases passed. - Green after the fix: `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1131 of 1131 tests. - `npx vitest run src/services/native-runtime/setup-readiness.test.ts` passes 98 of 98 tests. - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts` passes 246 of 246 tests with the real debug `paperclip-runnerd`. This includes the existing case where a replacement with a dispatched but unverifiable process still fails closed. - `npx tsc --noEmit -p server` and `pnpm --filter @paperclipai/paperclip-runner typecheck` pass. - Not run: a live hosted sandbox with a full disk. The staging data showed the exact error, but this change was not deployed there. ## Risks - Low risk. The change affects only failures that occur before the launch RPC. In that window, the launcher has not run any sandbox command that can start runnerd. - Retries now run for the reconnect grace window, which is bounded and uses the existing backoff. Each failed staging attempt cleans its own upload scratch. - This change does not free sandbox disk space. If the disk stays full, the run still fails, now with `transport_reconnect_grace_exceeded` and the ENOSPC detail. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use, shell execution, and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
367aa16800 |
fix(daytona): drop the stale session id on every sandbox restart (#19050)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs can execute inside a Daytona sandbox through the Daytona sandbox provider plugin > - The plugin runs every command for a lease in one persistent Daytona process session and keeps that session id in memory > - Daytona auto-stops an idle sandbox, and a stopped sandbox loses all of its process sessions > - Only the execute and resume hooks dropped the stored session id before a restart. Workspace realize, sync-in, sync-out, runner ingress, and interactive setup also restart a stopped sandbox, but they kept the dead id > - When host-side workspace preparation stays idle for longer than the auto-stop interval, sync-in restarts the sandbox, and the next command sends the dead id. The native run then fails with `native_session_interrupted: session not found` > - This pull request moves the session-id clear into the one shared start helper, so every restart path drops the stale id > - The benefit is that a run that waits through a long preparation opens a fresh session instead of failing ## Linked Issues or Issue Description **What happened?** Native runs on Daytona failed with `native_session_interrupted` and the error `session not found`. The failure came about 4 seconds after the provider session started. In every observed run, the workspace pack phase took longer than 15 minutes (917 s to 1442 s). This is longer than the default Daytona `autoStopInterval` of 15 minutes. **Expected behavior** After Daytona restarts an auto-stopped sandbox, the next command opens a new process session and the run continues. **Steps to reproduce** 1. Run one command on a Daytona lease, so the plugin opens and stores a session. 2. Let the sandbox stop (Daytona auto-stop after an idle gap). 3. Call workspace realize or sync-in on the lease. The hook restarts the sandbox. 4. Run the next command. It uses the stored session id, and Daytona answers `session not found`. No related open PR exists. The undeployed native fixes on master (#18992, #19025, #19026, #19004) do not touch the Daytona session store. ## What Changed - `ensureSandboxStarted` in the Daytona plugin now takes the lease scope. When the sandbox is not running, it clears the stored session id before it starts or recovers the sandbox. - All seven callers (execute, resume, realize workspace, interactive setup, runner ingress, sync-in, sync-out) pass their lease scope. The compiler enforces the scope on any new caller. - The execute hook no longer has its own copy of the clear. The resume hook keeps its clear because it also closes duplex channels before the restart. - Added three regression tests: a fresh session after realize restarts an auto-stopped sandbox, a fresh session after sync-in restarts it, and no new session when realize finds the sandbox already running. ## Verification - Red before the fix: `pnpm exec vitest run src/plugin.test.ts -t "auto-stopped sandbox|already running"` in `packages/plugins/sandbox-providers/daytona`. The two restart tests failed with `expected "vi.fn()" to be called 2 times, but got 1 times`. The running-sandbox safety test passed. - Green after the fix: the full Daytona plugin suite (`pnpm exec vitest run`) passes, 542 passed and 8 skipped. - `pnpm typecheck` in the Daytona plugin package passes. - Not done: a live Daytona run that idles past the auto-stop interval. The fix needs deployment before the staging failure can be checked again. ## Risks - Low risk. The clear runs only when the sandbox is not in the `started` state. A stopped sandbox has no live session, so no live shell can leak. A running sandbox keeps its stored session, and a test covers this case. - The recover path for a recoverable `error` state now also drops the id. A recovered sandbox restarts, so its old session is also gone. - Duplex channels on a restarted sandbox are still closed only by the resume hook. This PR does not change that behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
a0c0851766 |
Runner: add private host control channel (#19129)
## Thinking Path > - Paperclip coordinates AI agents and their execution. > - A hosted Runner can remain alive between turns. > - Its host needs to park, claim, or retire that process without a persistent Core connection. > - Those requests need a private channel with bounded input and safe retries. > - This PR adds the channel codec and socket handling as an isolated foundation. > - Later lifecycle changes will validate and execute the requests. ## Linked Issues or Issue Description **Current behavior** The Runner has no task-private host control channel. The earlier combined lifecycle change was split for review. **Proposed behavior** Add a private Unix stream protocol for host lifecycle requests. This PR does not wire it into Runner startup or advertise support. It can be reviewed independently of stopped-process recovery. **Reason and benefit** Review frame parsing, credential handling, and retry behavior before adding park and claim transitions. This replaces the transport portion of #19060. The source-recovery work supersedes #19057 and #19058. ## What Changed - Add `host_control.rs` and enable the existing rustix dependency's socket APIs. - Accept only a connected Unix stream. Set close-on-exec so provider children do not inherit it. - Bound request frames to 16 KiB. Reject unknown fields, malformed requests, and EOF with a credential-free error. - Add socket-pair tests for framing, deadlines, retries, conflicts, and receipt-window reset in a separate test file. ### Transport contract Each request is one newline-delimited JSON object. Each response is `{ "id": "request-id", "status": "..." }` followed by a newline. | Request | Required binding | Intended effect in the later lifecycle consumer | | --- | --- | --- | | `park` | Six Runner identity fields, last controller command sequence, acknowledged source sequence | Retain the process and mounts after the controller has settled that exact cut | | `claim` | Park request ID and a fresh bootstrap ticket | Reconnect the retained process through authenticated bootstrap | | `retire` | Park request ID; optional claim ID and identity/sequence cut | Retire only the matching parked or claimed process | The six identity fields are Runner instance, environment lease, run, normalized session, turn, and item IDs. This module only parses requests and sends responses. The lifecycle consumer must validate these bindings before performing an effect. Before applying a request, the consumer checks the in-memory response cache. An identical retry receives its prior status. Reusing a completed request ID with different content returns `conflict`. Conflict responses do not reserve an ID. A successful `parked` response starts a new cache window; the consumer's identity, sequence, and park-ID checks must reject requests from older windows. Credentials are neither persisted nor included in errors. Active reads are nonblocking. Parked reads block on socket input and may have a caller-supplied receive deadline. Partial frames do not extend that deadline. No idle polling is added. ## Verification - `cargo test --offline --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --lib host_control::` — 6 passed. - Confirmed the repeated-chunk test fails when reads incorrectly reset the original deadline. Restored production code and reran the targeted suite. - The initial policy gate did not recognize inline Rust tests. Moved the same tests to the existing `src/tests/` convention. - GitHub CI passed on `aaac996aba76`. Greptile is 5/5 on that commit, with no unresolved review threads. - A local merge-tree check against current master `1a7eb70a37bf` is conflict-free. - Reviewed changed lines for secrets, personal data, and private deployment references before push. ## Risks The module is dormant in this PR. It does not make a VM sleep, detach Core, or evict a Runner. Integration must enforce the documented identity and sequence fences before replying with success. The response cache is process-local and does not provide recovery across process death. Linux and Windows compilation are covered by CI; local socket tests run on macOS. ## Model Used OpenAI Codex with tool use and code execution. GPT 6.1 Sol with High effort handles the parallel recovery work; this isolated channel change is implemented by the coordinating Codex agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
aa8372fd5d |
fix(accounting): post late usage to its owning run (#18991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so Paperclip accounts usage per run. > - Codex can report more usage for a turn after its run's charge was posted. > - The Runner then marked the session `reconciliation_required`. If another run was attached, it also failed that run. Core refused the delivery with `native_usage_after_finalization_requires_reconciliation`. > - That usage has already happened. Blocking the session cannot undo it, and the debt waited for an operator while the run totals stayed wrong. > - Codex totals are cumulative, and the Runner already knows the owning run in most cases. So the correct amount can be posted automatically. > - This pull request reports the exact increment with its owner, and posts it to the owning run as a separate cost event. > - The benefit is that run accounting becomes correct without stopping work, and an old run's usage can no longer fail a newer run. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for "late usage", `native_usage_after_finalization_requires_reconciliation` and `codex_usage_accounting_integrity`. I found no duplicate. **What happened?** Code review found this while diagnosing session recovery failures. When Codex reported usage above the known total for a turn that was not active, the Runner blocked the session. If that turn belonged to a previous run, the Runner also failed the attached run with `codex_usage_accounting_integrity`. Core kept the receipt as debt, and could correct it automatically only if the token counts were unchanged. **Expected behavior** Late usage does not stop any run or session. The charge goes to the run that used the tokens, and it is auditable. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session and attach the next run. 2. Have Codex send `thread/tokenUsage/updated` with a higher total for the finished turn. 3. On the unfixed code, the attached run fails and later attaches are refused. `late_codex_usage_requires_durable_reconciliation_before_or_after_handoff` encoded this behavior. **Paperclip version or commit** `master` at `04a24f4e06`. **Deployment mode** Any deployment that uses a native Runner with Codex. ## What Changed - **Runner:** late usage no longer sets `reconciliation_required` or fails a run. For a turn that belongs to a previous run, the Runner emits `usage.reported` with the increment in `runDelta` and the owner in `reconciliation.priorOrigin`. `runDeltaAvailable` stays `false`, so an older Core does not bill it to the attached run. - **`CodexRunUsage::absorb_late_usage`:** treats only usage above the known thread total as late, and keeps the attached run's delta unchanged. A report at or below that total adds nothing, so the sum across runs always equals the thread total and a replay cannot charge twice. - **Delivery:** `captureLateUsageForOwner` stores the increment as a late receipt on the owning run. The owner must be a finished run in the same company and native session. If the Runner cannot name the owner, Core uses the session's last finished run and records the attribution as `inferred`. - **Local Runner:** late usage stays with the run that is active when it arrives, or the next run. Usage reported between two runs stays in the next run's delta, so the total stays exact. - **Accounting:** `reconcileLateRunUsageReceiptInTransaction` posts a token increase as a new cost event with the key `heartbeat:<run>:late:<receipt>`. It updates the run totals so the accounting integrity check still agrees, records `cost.late_usage_posted` in the activity log, and queues budget evaluation. - **No more refusal:** `persistUsageReceipt` no longer throws after it saves late evidence, so late usage cannot wedge a Runner outbox. - **Covered receipts:** an older same-source receipt at or below the accepted totals, with matching billing, is cleared because a newer cumulative receipt already covers it. Conflicting billing stays as debt. - **Older Runners:** a zero-filled invalidation stays on the delivering run. - **Unchanged:** a newer lower total, a changed model or billing type, and per-model receipts stay as visible accounting debt for an operator. - **Unchanged:** an unexpected provider message during the warm attachment drain still fails closed. A separate change addresses that gate. - **Docs:** added "Late provider usage" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 99 passed, 2 ignored. - `cargo test --locked -p paperclip-runner-core --lib codex_usage`: 7 passed. This includes new tests for the out-of-order and repeated report cases. - `cd server && pnpm exec vitest run src/services/run-cost-accounting.test.ts src/services/native-runtime/native-usage-delivery.integration.test.ts src/services/native-runtime/native-recovery-usage-delivery.test.ts src/__tests__/cost-data-quality.test.ts`: passed against embedded PostgreSQL before the last review fixes, and the accounting and delivery files were re-run after them. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover exact and inferred owners, repeated delivery, late usage that arrives before the owner's charge posts, another session's run, an older Runner's report with no increment, a same-run increase, and a lower total. They also run the accounting integrity check after posting. - **Review:** Greptile Apex scored 1/5, 4/5, 3/5, 2/5 and 4/5 while findings were fixed, then 5/5 at the current head. - **Local Runner:** driver tests cover usage reported between two runs, including across a restore. No end-to-end local run has exercised it. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** a read-only check of a test deployment found no late receipts in its history, so this path has only run against fixtures. ## Risks - **Run totals can rise after a run finishes.** This is deliberate. The correction is a separate cost event and an activity log entry. - **Budgets are corrected after the fact.** A late charge can put an agent over budget after later work has started. Budget evaluation runs after each correction, but it cannot prevent that work. - **Local sessions accept imperfect attribution by design.** Their late usage can land on the active or next run, because local sessions have no durable path for routing it. This also applies after a Stop or a pause. The total stays exact. - **Out-of-order reports can leave tokens on the wrong run in the session.** If a prior turn's report arrives after the attached run has already counted past that total, those tokens stay on the attached run. Moving them would need a lower correction to the attached run. The total stays exact. - **Inferred attribution can name the wrong run in the session.** The usage still belongs to that session and the total stays correct. The attribution is recorded as `inferred`. - **A session with no finished run charges the delivering run,** as an inferred late charge after its own charge posts. - **Rollout order:** deploy Core before the Runner. An older Core would keep this event on the attached run as incomplete usage, and would still refuse a same-run late receipt. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
15114bb9a0 |
fix(workspaces): keep durable seed copies above the free-space reserve (#18999)
persistDurableSeedArchive now checks the source file and applies the shared workspace free-space reserve before copying, so a large seed cannot fill the instance volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
8002dfb06e |
fix(evals): repair bounded TestOps protocol admission (#19011)
## Thinking Path > - Paperclip lets people manage AI agent work through a governed control plane. > - Runner protocol qualification uses real providers and records selected cases in TestOps. > - A manual registration run starts an Allure watch session before it validates the case selection. > - Watch creates the job run, but it does not fill the empty job-run input in the child environment. > - The selection helper therefore stops before it can validate the new job run. > - A configured AWS concurrency ceiling can also reject a valid smaller hosted-runner request. > - Hosted tool-cache permissions can also violate Runner qualified-executable admission before a provider starts. > - This pull request captures official session identities, keeps both concurrency ceilings, and stages the same Node bytes in an owned directory. ## Linked Issues or Issue Description Refs #18985. **What happened?** The [first registration attempt](https://github.com/paperclipai/paperclip/actions/runs/41877603952) created an Allure job run, then stopped with `runner_testops_jobrun_missing`. The child inherited an empty `ALLURE_JOB_RUN_ID`. The helper made no job-run API read and executed no provider case. The current configured concurrency ceiling is 100. The controlled hosted cap is 57. The original catalog guard rejects 100 before it applies a valid requested value of 2. The [first service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41884365232) reproduced this second pre-provider failure. Its selected case was correctly reported as broken. No provider cell ran. A [second service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41887671585) passed selection, lock and cost preflight. Runner rejected a qualified ACPX launch artifact as group- or world-writable. It recorded no model request. The verified sidecar is mode 0755. Hosted Node is the only other launch artifact, which points to Node as the cause. The original run did not retain its exact mode. The TestOps reporter retained the infrastructure failure. **Expected behavior** The manual path must capture the job-run and launch IDs from the active official watch session. It must validate those IDs against the same GitHub run, job, launch and standing plan before provider work. A requested concurrency value must fit the configured ceiling and the selected runner cap. The default must use the lower ceiling. Before credentials, the worker must stage the same Node bytes in an owned directory and preserve Runner permission and digest admission. **Steps to reproduce** 1. Dispatch the trusted Runner protocol workflow with reporting enabled and an empty job-run service input. 2. Let `allurectl watch` create the registration session. 3. Observe that the original helper rejects the empty child input before selection validation. ## What Changed - Call the supported `allurectl job-run env` command for manual registration. Use a fixed argument list without a shell and bound its time and output. - Parse only numeric job-run and launch IDs. Reject missing, duplicate, malformed or conflicting values. Discard all other command output. - Bind the captured launch ID to the authoritative job-run response. Keep the service-supplied ID path and its existing checks. - Apply requested parallelism within both configured and physical ceilings. Use their minimum as the default. Keep the selected-case capacity and provider budget checks. - Snapshot the same current Linux Node bytes into a fresh owner-only directory before credentials. Verify the copy, execute it to bind its path and version, and put only that directory first on the job-local path. Record original and copied Node modes and hashes plus the actual sidecar mode and hash. Keep shared cache permissions and Runner security checks intact. - Add focused context, identity, concurrency, permission and secret-exclusion controls. Supply the production physical cap in the existing release-workflow fixture. - Fix the existing Kimi cancellation fixture to await the complete readiness sentinel within the same 100 by 10 ms bound. Keep its cancellation and cleanup assertions. ## Verification - The original empty-ID failure reproduces locally on the actual helper with all 41 standing cases and zero job-run API reads. - All 63 Node publication, projection, metrics, permission and workflow-security tests pass. The repaired manual context flows through the actual selection helper to case 194. Wrong launch, duplicate IDs, malformed values and command failures still stop selection. - The original catalog prefix rejects configured 100 / hosted 57 / requested 2. The repaired prefix admits 2, clamps a missing request to 57, keeps legacy 100, and rejects requests above either ceiling. - The full release-policy CI command passes locally: 72 release-workflow tests and 107 companion tests. The first sandbox run could not bind its loopback fixtures; the same tests passed with local-network access. The prior-head CI fixture omission is retained and fixed by supplying the same physical cap as production. - Both full static Node file groups pass locally: 418 passed and 11 existing opt-in skips, at concurrency 2. Initial runs used stale dependency links and failed. Correcting only isolated ignored links to declared cached dependencies and pnpm 9.15.4 removed those setup failures. Local Node 26.4 differs from CI Node 24; the exact-head CI below provides canonical verification. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - The new staging helper passes all four tests as a nonroot Linux user with networking disabled. An actual retained Runner comparison reproduces the qualified-artifact rejection with Node mode 0777. The same Node bytes at 0500 in a 0700 directory pass that launch boundary and reach a closed provider session. The local turn still fails without credentials or network; this is permission proof, not a protocol case pass. Local Linux Node 24.21 differs from the hosted 24.22.1 executable. - Exact final head `8e06ee5611` [repository CI](https://github.com/paperclipai/paperclip/actions/runs/41892896405) completed successfully. Typecheck, build and all four Runner lanes passed. Runner package tests passed 4950 tests in 297 files, with 10 existing skips. Fresh exact-head Greptile review is 5/5 with no actionable findings or open threads. Prior `59378f` CI Kimi empty-sentinel race and canceled chat worker remain retained. - The patched manual workflow has not run against live TestOps. A separate service-triggered probe supplies its own ID. The service probe used the original concurrency guard with a temporary configured ceiling of 57. After the probe, the original repository ceiling 100 was restored with no active campaigns. That probe does not verify this patched concurrency code. The next launch waits for the protected merge and source/lock verification. ## Risks An incompatible official CLI output fails closed. The helper never evaluates the output or records its unknown variables. The existing job, workflow URL, source, plan and provider checks remain required. An omitted concurrency request now uses the lower of the configured and physical ceilings. Explicit requests above either ceiling still fail. The worker now executes a private copy of its existing Node binary. A copy, mode, hash, path, version or sidecar mismatch stops before credential access. The copy does not make dynamically linked Homebrew Node portable; this workflow path is Linux only. This change does not alter the provider roster, policy, limits, retries or result grading. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, code editing, repository inspection and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Nicky LeachandClaude Opus 5.5
|
be20d4ad97 |
fix(projects): keep projects on the storage chosen at creation (#19036)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project stores its work in Core/local storage or in a storage plugin > - Today, Core finds the installed default each time it reads a project > - So installing a default storage plugin moves existing projects, without a saved change > - Agents that cannot reach the new storage then lose access to new workspaces > - This pull request makes Core apply the default once, when it creates a project > - The benefit is that a project stays on its storage until a person moves it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. **What existing behavior does this improve?** How Core chooses the storage for a project when a plugin claims `defaultForProjects`. **Current behavior** Installing such a plugin changes the storage of every existing project with no explicit choice. Disabling the plugin changes it back. Nothing is saved, so the project has no fixed owner. Execution environments and storage are separate contracts, so an agent can lose access to new workspaces on a project that it used before. **Proposed behavior** Core saves the storage choice on the project when it creates the project. A project with no saved choice uses Core/local storage. A plugin-backed project stays attached to its plugin. A Core/local project can move to a plugin. **Reason and benefit** A project stays on its storage until a person moves it, so installing or disabling a plugin does not change where work goes. **Alternatives considered** - Keep the default at read time and add a process that saves a choice on existing projects when a plugin arrives. This adds a background step and a race with new projects. - Add a new database column. The existing policy field already holds the choice, so a migration is not necessary. ## What Changed - `resolveProjectWorkspaceProvider` reads only the saved choice. It no longer applies an installed default. - `withDefaultProjectStorage` applies the default when Core creates a project, including a personal private project. With more than one default, a project with no explicit choice is refused. This is the existing conflict rule, now applied at creation. - `projectService.update` refuses a change of storage on a plugin-backed project. An update that omits the field keeps the attachment. - Plugin lifecycle delivery sends project events to a storage provider only for projects attached to it, and for deleted projects. Core acknowledges the other events for that provider. Other plugins continue to receive all project events. - The project settings control has no "Inherit" option. It is locked on a plugin-backed project. - `ProjectWorkspaceProviderConfiguration.source` no longer has the `"default"` value. - A project with no repository or folder ignores its saved storage choice when a task runs. A storage plugin cannot manage this type of project, so its tasks use a plain task directory. Without this change, the second run of a task reset the agent session, and remote agents could not use an empty task directory. - The plugin SDK README and the plugin authoring guide describe the new rules. ## Verification - `npx vitest run server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/plugin-lifecycle-delivery.test.ts server/src/__tests__/plugin-workspaces.integration.test.ts packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass. - `npx vitest run src/components/ProjectProperties.concurrency.test.tsx` in `ui/`: 9 tests pass. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/ui typecheck` pass. - `npx vitest run` on `heartbeat-session-preparation`, `workspace-preparation`, `remote-workspace-source`, `project-workspace-provider.integration`, and `plugin-workspaces.integration` in `server/`: 231 tests pass. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - **Behavior change at upgrade:** an existing project that uses a plugin only through the installed default, with no saved choice, moves to Core/local storage after upgrade. Its plugin files stay in place, but new work does not use them until an operator sets `storageProviderPluginId` on the project. This pull request has no automatic backfill. - **Missed lifecycle events:** a provider does not receive events for a Core/local project. If the project moves to that provider later, the update event starts its reconcile. - **One-way move:** a person cannot move a plugin-backed project back through the API or the UI. This is intended. If a plugin is disabled, the project fails closed, as it does today for an explicit choice. - This pull request does not stop a person from assigning an agent to a project with storage that the agent cannot reach. That is follow-up work. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
8a0e7ab511 |
fix(runner): preserve ordinary append for unscoped governed terminals (#19044)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can park on a saved human question or approval. > - Its terminal event can return a committed receipt before observers finish. > - That option requires an explicit turn ID at the event port. > - The protocol also permits events without a turn ID. > - The runner requested the option for those events, which the port rejected. > - This change keeps those events on the ordinary append path. ## Linked Issues or Issue Description Refs: #19025. **What happened?** The native governed-wait observation can accept a schema-valid `item.completed` without a turn ID. If the later terminal also has no turn ID, the runner compares two absent IDs and requests `deferPostCommitObservers`. The event port rejects that option. The run then fails with `governed_wait_provider_terminal_unproven`. **Expected behavior** Only a terminal with an explicit turn ID can request the early receipt. An unscoped terminal must keep the ordinary append behavior and await its observers. The event port must retain its turn identity check. **Steps to reproduce** 1. Create a pending human-only interaction from the run. 2. Pass a schema-valid completed tool event without a turn ID through the real governed-wait observation. 3. Supply an interrupted terminal without a turn ID during the cancellation drain. 4. Run the actual native runtime with the PostgreSQL event port. The parent code rejects the deferral request and reports a failed terminal. **Paperclip version or commit** Reproduced on `4ea53f9eb5a24de10db3dfda18456132987c4af8`. The follow-up was identified by a review published after #19025 merged. No observed provider emission or original-task recovery is claimed. I checked current master and related PRs before this correction. **Deployment mode** Built from source. The regression uses real PostgreSQL and a deterministic provider boundary; it makes no model call. ## What Changed - Require a truthy terminal turn ID before the runner requests deferred observers. - Keep ordinary append behavior, the port identity check and all timeouts unchanged. - Exercise the production governed-wait observation with a persisted pending interaction and the real event port. - Retain exact-turn observer and checkpoint controls. Add a direct rejection control for unscoped deferral. - Document that unscoped terminals await their observers. ## Verification - First meaningful red: the actual observation, native runtime and PostgreSQL port fail on the unmodified parent runtime. The new case reports `NativeProviderTerminalFailure` in 143ms; this is not a setup or deadline failure. - Focused regression and authority controls: 12 passed, 69 filtered. Full runtime suite: 203 passed. Full PostgreSQL port and timing suites: 84 passed. - The tests leave the human-only interaction pending with continuation policy `none`, accept one result, and start only one turn. Existing scope, invalid evidence, checkpoint, abort, duplicate and restart controls remain. - Exact candidate: `8840ea2c9c308e5a60423e67fb3612d24029ba84`. Repository-wide `pnpm -r typecheck`, `pnpm build` and `pnpm check:module-boundaries` passed on this exact head. `pnpm test:run` is still in progress and is not reported as passing. Current-head CI passed: 55 success, 3 skipped and 1 neutral, including required `ci / verify` and `ci / e2e`. Greptile and Ponytail both reviewed this exact head at 5/5 with no blocking findings. The native review was source-only. No unresolved review threads remain. - No UI, database schema, agent permission or mapped user entry point changes. No deployment, live retry or provider replay was performed. ## Risks Unscoped terminals retain the existing observer wait and can still fail if it exceeds the existing drain deadline. They do not gain an early receipt. The change does not supply a missing turn ID or weaken source, company, run, session, cancellation or transaction checks. A PRP terminal is not physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c99bcd89fe |
fix: preserve Product TestOps evidence and CI readiness (#19000)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the app and Runner. > - A trusted reporter sends retained results to TestOps. > - Free-form metadata and a short upload observer can lose safe, valid evidence. > - UUID navigation can also discard task metadata before the canonical query returns. > - Hermetic CI files start child processes that need CPU capacity within their readiness limits. > - This change closes the reporting contract, retains authenticated core aliases, and reserves CI CPU capacity. > - Original case identities, conditions, assertions, verdicts and deadlines remain intact. ## Linked Issues or Issue Description Refs #18983. This addresses its late [metadata disclosure finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470789114) and [upload readiness finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470790366). **What happened?** The reporter accepted model IDs and matcher names through syntax filters. A secret-shaped string could pass those filters. First-task results also use their top-level model field for an observed alias. Treating it as configured metadata can reject valid retained evidence. The upload observer stopped after 22 seconds of waits. [Required CI on the earlier head](https://github.com/paperclipai/paperclip/actions/runs/41886885193) also found two real browser title failures and five native package readiness failures. A deterministic control reproduced metadata loss during UUID-to-identifier navigation. A separate fixed-resource control reproduced startup deadline failures with four concurrent hermetic fixtures, while one and two fixtures passed. **Expected behavior** Reject unknown transmitted metadata before creating upload files. Preserve every original matcher verdict and attempt receipt. Observe server processing without rerunning the product. Keep authenticated task metadata visible through canonical navigation. Reserve capacity for hermetic child processes without changing the original readiness deadlines. **Steps to reproduce** Use a secret-shaped model ID or cleanup state in retained evidence, return an observed first-task model alias, or let upload processing settle after 40 seconds of virtual time. Hold the canonical core query pending after an authenticated UUID read. For the CPU control, run the same providerless startup/recovery fixture alone, four times concurrently, and twice concurrently under a fixed 4-vCPU quota and its original 8-second readiness limit. ## What Changed - Bind configured model receipts to each exact declared case model. Four first-task cases use captured agent configuration; their observed aliases and fallback text remain digest-only. A missing capture stays null / Not recorded. The other 16 cells use their configured profile model. Reject wrong or unknown configuration before any upload file exists. - Keep the existing model distribution: 18 Codex cells on `gpt-5.6-sol`, configured-repository case 9 on its existing `gpt-6.1-sol` override, and task-workspaces case 54 on `openrouter/deepseek/deepseek-v4-flash-0731`. The repository override exists at the [original canary source](https://github.com/paperclipai/paperclip/blob/86814c6550f52e99ede150a883bdc773cbb540e3/tests/runner-e2e/catalog.ts#L1148). No model configuration changes. - Allow only existing Matcher kinds, closed failure classes, and cleanup states `passed`, `failed`, or `not_started`. Publish each matcher's kind, ordinal, original boolean verdict and SHA256 path receipt. Full conditions remain in the linked original workflow artifact; raw dynamic path bytes do not enter TestOps. - Observe the same upload for at most 55 seconds of poll waits. Product scenarios, provider retries and scenario deadlines do not change. - Retain task core data under only the authenticated response's own UUID and identifier core keys. Preserve existing canonical data. Keep arbitrary references and full-detail keys empty until their own reads. - Retain a first-failure trace for the existing Cloud recovery test. Keep its assertions, timeout, retries, mock data and artifact policy unchanged. This adds diagnosis evidence; it does not claim to fix that separate Cloud title timeout. - Reserve half the available CPUs for Runner package CI workers, with the existing upper cap of four. A 4-vCPU host uses two workers; an 8-vCPU host still uses four. Local defaults and test selection do not change. ## Verification - Reporting controls first reproduced the two original gaps. All 14 Node contract tests now pass. Unknown configuration creates neither a result directory nor an export summary. A model valid for a different selected case is also rejected. Original functional failures and earlier accounting receipts survive. - The catalog compatibility test passes for all 20 selected cells, exact model tuples, real pure grader outputs and actual `projectFirstTaskResult` shapes. It covers observed aliases, fallback text, absent evidence and wrong or secret-shaped configured metadata. These are contract controls, not live journey passes. - Virtual-clock controls accept a correct result that settles at 40 seconds and reject a persistently wrong verdict after 55 seconds. They make no product or provider calls. - Two new core-cache controls failed before repair. The query/cache suites now pass all 8 tests. They hold canonical refresh pending and verify title/company retention, newer canonical data, full-detail enrichment and unrelated keys. UI typecheck and token gates pass. - The unchanged exact UUID cross-company link/upload browser test passed once after repair: 5.8-second scenario, 32.0-second complete command. The unchanged exact Cloud task-entry diagnostic passed once: 5.4-second scenario, 23.3-second complete command. Its original canonical-only timeout remains unexplained. All five Cloud tests still collect after the trace setting. - The providerless Linux control used the retained Runner and compiled controller from source `3a2249dd`, with a fixed 4-vCPU quota and original 8-second readiness limit. Solo passed 1/1; four concurrent fixtures failed 4/4; two concurrent fixtures passed 2/2. At each failure, Rust had accepted command 2 but startup was still pending; checkpoint and published status agreed, one connection was authenticated, no replay backlog existed, and the sidecar had not launched. Passing controls preserved the expected startup failure through controller replacement. No network or credentials were available. - That control uses Linux amd64 emulation and Node 24.21; hosted CI used Node 24.22.1. It supports the worker-cap change, but does not prove that the original `m5n.xlarge` host caused every CI failure. The original red attempts remain recorded. - Actual config-load controls verify 1 CPU → 1 worker, 2 → 1, 4 → 2, 8 → 4 and 16 → 4, with no local, timeout or retry override. The isolated policy change preserves the same 297 test-file inventory; integration of current master `89703c3ff7` retains all 298 current native package files. The original PR package/native command plans remain unchanged. - Final head `877e28da32f2d2f0d12906c89aff085e2c6ea943` passes both required checks, `ci / verify` and `ci / e2e`, in [CI run 41897677652, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41897677652/attempts/2). The native package passes 4,950 tests with 10 existing skips. All eight browser shards pass; the Cloud shard passes all 41 tests, including all four unchanged recovery cases. - Attempt 1 preserved 46 successful jobs but its one-file heartbeat recovery shard hit the original 20-minute job limit without a named assertion or completion verdict. Its cause remains unidentified. One explicitly bounded debug replay retained those 46 successes and ran only that leaf and its dependent rollup. The unchanged file passed all 412 tests (346.91-second file, 371.34-second Vitest command, 7m11s job). Source, cases and deadlines did not change. This establishes intermittency, not a causal repair. The original timeout and debug pass both remain recorded. - Fresh [Greptile 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637604918) and [Ponytail 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637489454) reviews name exact head `877e28`. The single review thread is resolved. Live TestOps bootstrap and complete 20-case acceptance remain held until reviewed delivery. CI and focused controls do not count as product plan passes. ## Risks - Future model or Matcher kind changes require a reviewed reporting-contract update. - TestOps shows a closed Matcher kind and ordinal. The path digest binds the original condition in retained evidence. - Core alias data retains the existing authenticated query-client and server company boundaries. It never becomes full-detail evidence. - Fewer CI workers can increase one job's duration. The full suite must verify the final policy. The paid evaluation matrix and its parallelism do not change. - The original Cloud title timeout and the first-attempt heartbeat CI hang do not have proven causes. Current unchanged assertions pass, but this does not establish repeatability. The Cloud test now retains its first-failure trace. A successful upload or focused control alone cannot establish a complete product pass. ## Model Used OpenAI Codex with reasoning, tool use and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3cdb367369 |
fix(acpx): report how the sandbox agent process session ended (#16366)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. The agent process
runs in a sandbox, and a host-side process session bridge carries the
ACP stream.
> - Some hosted turns fail with only "The connection to Claude was
lost." (`acpx_turn_failed`, terminal session category `connection`). The
failure report shows one pending `execute` tool. It does not show why
the agent process went away.
> - The bridge already records how the session ended: the remote exit
code and signal, or a host proxy close. But it writes that evidence only
to the run log. The run failure report never receives it.
> - Without it, we cannot tell an OOM kill (`SIGKILL`) or a provider
process exit from a bridge or proxy close. Each cause needs a different
fix, so we must not guess.
> - This pull request keeps the first terminal evidence on the bridge
handle and copies it, as closed values, into the run failure report's
execution context.
> - The benefit is that the next recurrence names the cause, and the
correct fix can follow.
## Linked Issues or Issue Description
**What happened?**
Hosted `claude_local` runs on staging and production failed with "The
connection to Claude was lost." One run failed about 4 minutes in, 7–10
seconds after its last ACP event. Another failed after about 3.5 hours
with no ACP events for most of that time. Both had one pending `execute`
tool. The report had no information about how the sandbox agent process
ended.
**Expected behavior**
The run failure report says how the sandbox agent process session ended.
Examples: the remote process exited with a code, the remote process was
killed by a signal, or the host proxy closed.
**Steps to reproduce**
1. Start a hosted ACP `claude_local` run on a sandbox target.
2. Kill the agent process in the sandbox while a tool runs, for example
with `kill -9`.
3. See that the run log has an `ACP process session terminal:` line, but
the run failure report has no matching field.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
22735cfcea |
fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description **What happened?** A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. **What did you expect to happen?** A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. **Steps to reproduce** Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
ff7e7b4963 |
fix(claude-local): make sandbox tool commands the first OOM victims (#16844)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted `claude_local` runs execute in a sandbox. The sandbox has a fixed memory limit (4 GiB in the observed sandboxes). > - In that sandbox, every process has the same `oom_score_adj`. When a Bash tool command uses most of the memory, the kernel kills the largest process. That can be the Claude Code process. > - When Claude Code is killed, `claude-agent-acp` reports "The connection to Claude was lost." and the whole run fails. Server logs show `Claude Agent process died: Claude Code process terminated by signal SIGKILL` for each failure. > - A failed tool command is recoverable. A dead agent is not. > - This pull request makes Claude's tool commands the first out-of-memory victims in managed sandboxes. > - The benefit is that a memory-heavy command fails with exit 137, and the agent continues the run. ## Linked Issues or Issue Description Refs #16366 (diagnostic PR for the same failure). Related: #14509 protects the host control plane from local agent processes. That PR does not cover a sandboxed agent and its own tool commands. **What happened?** Hosted `claude_local` runs fail with `acpx_turn_failed` and "The connection to Claude was lost." while one `execute` (Bash) tool call is pending. The pending commands are memory-heavy, for example `tsc --noEmit` with a 3.3 GB heap, a monorepo `pnpm install`, or a type check plus `vitest`. At the same second, the server log shows `Claude Agent process died: Claude Code process terminated by signal SIGKILL`. Over 10 days, all 12 such lines on one staging stack were `SIGKILL`, and they match the failed runs. The sandbox memory cgroup has `memory.max` 4 GiB and `memory.oom.group` 0, so the kernel kills one process. Every process has `oom_score_adj` 0, so the kernel picks the largest one. **Expected behavior** Under memory pressure, the kernel kills the tool command. Claude gets a failed command (exit 137) and continues the run. **Steps to reproduce** 1. Start a `claude_local` run on a sandbox target with a 4 GiB memory limit. 2. Ask the agent to run a memory-heavy command in a large repository, for example a full `tsc --noEmit` and a test run together. 3. The kernel kills the Claude Code process, and the run fails with "The connection to Claude was lost." **Agent adapter(s) involved** Claude Code (`claude_local`, ACP lane and CLI lane, managed sandbox targets). ## What Changed - `claude-config.ts`: add `CLAUDE_TOOL_SHELL_PREFIX_SCRIPT`. Claude Code calls a `CLAUDE_CODE_SHELL_PREFIX` with one argument, the full command string. The script raises its own `oom_score_adj` to 1000 (no privilege is necessary to raise it). Then it runs the command string in the shell that Claude Code selects: `CLAUDE_CODE_SHELL`, then `SHELL`, when either is an executable bash or zsh, else zsh or bash from `PATH` (bash first when `SHELL` names bash). - `buildRemoteClaudeConfigMaterializationCommand` can install the script (umask 077, write to `.partial`, `chmod 700`, `mv`). The install runs last in the existing materialization command, so it adds no sandbox round trip. It prints a marker only when the file is in place. - `materializeRemoteClaudeConfig` returns `toolShellPrefixInstalled`. It is true only when the command exits 0 and prints the marker. - ACP lane (`acp.ts`) and CLI lane (`execute.ts`): set `CLAUDE_CODE_SHELL_PREFIX` to the installed script only when the install is confirmed. A prefix path that does not exist would fail every Bash tool call, so a failed install keeps the previous behavior. An operator `CLAUDE_CODE_SHELL_PREFIX` in the adapter env is kept, and no script is installed. - `docs/adapters/claude-local.md`: add "Sandbox memory limits". ## Verification - `npx vitest run src/server/claude-config.test.ts` in `packages/adapters/claude-local`: 10 pass, 6 new. The new tests install the script with the real materialization command through `/bin/sh` and check: - the file content, mode 0700, the marker, and that no `.partial` file remains; - the command runs with `oom_score_adj` 1000 while the caller keeps its own score (Linux only); - shell selection: a valid bash `SHELL`, an invalid `CLAUDE_CODE_SHELL` that falls back to `SHELL`, and a `SHELL` of `/bin/sh` that falls back to bash from `PATH`; - quoting and the exit status of the command are kept; - an operator prefix wins, and no marker means no install. - `acp.test.ts`: the managed-config test now checks that `CLAUDE_CODE_SHELL_PREFIX` names an executable 0700 file next to the config dir. A new test checks that an operator prefix is kept and no script is installed. Both use the local sandbox runner, which runs the setup commands for real. - `tsc --noEmit -p packages/adapters/claude-local` is clean. - Two tests fail in my sandbox both with and without this change: `uses the same default model in ACP startup and session identity` and `reselects the full assignment and bootstrap guidance after a failed resume`. They also fail on clean `master` in the same environment, so I think they depend on that environment. CI covers them. - Real Claude Code check (Claude Code 2.1.286): with `CLAUDE_CODE_SHELL_PREFIX` set to the script, a Bash tool call printed `oom_score_adj` 1000 and ran in zsh (the selected shell), and its parent was the `claude` process at score 0. Without a prefix file that exists, every Bash tool call fails with exit 127. This is why the env is set only after a confirmed install. - Real OOM check in a 4 GiB Daytona sandbox (`memory.oom.group` 0). A stand-in "agent" process held 1.2 GB at score 0, and a tool command started 14 processes of 250 MB each: - without the prefix, the kernel killed the stand-in agent (SIGKILL, 137; `oom_kill` +1); - through the installed prefix, the stand-in agent stayed alive, and the kernel killed 5 tool processes (`oom_kill` +5). - Assumption that could invalidate the design: Claude Code passes one argument (the command string) to the prefix and honors the variable through `claude-agent-acp`. I checked the argument shape with a logging prefix, and `claude-agent-acp` 0.73.0 spawns the CLI with `{ ...process.env }`. ## Risks - Every Bash tool command and hook in a managed sandbox now runs through one more `sh` process and an `exec`. The cost is a few milliseconds for each command. - If a future Claude Code release changes how it calls `CLAUDE_CODE_SHELL_PREFIX`, Bash tool calls can fail. The tests above and the documented variable limit this risk. Removing the variable restores the old behavior. - The inner shell is not a login shell. Claude Code sources its own shell snapshot inside the command string, so the environment stays the same. If Claude Code runs without a snapshot, it uses a login shell. Only exported variables then pass through to the inner shell. - If the selected shell differs from the one that Claude Code chose (for example, a custom `CLAUDE_CODE_SHELL` that is not bash or zsh), the script uses the same fallback order as Claude Code. - This does not help when the Claude Code process itself uses most of the memory. The local (non-sandbox) lanes do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and code execution. It read the repository and the Claude Code CLI behavior, ran the OOM experiment, edited the code, and ran the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (two tests fail on clean `master` in my sandbox too; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
e107a75755 |
fix(native): decide session settlement from live state (#18992)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so a later run attaches to a session that an earlier run settled. > - Three gates decide if a session is settled: Core's journal check, the Runner's attach check, and the Runner's drain of provider messages after a turn. > - Each gate kept a list of harmless events and refused anything else. Every new harmless event became an outage. > - An example: Codex replays a settled turn's usage on `thread/resume`, so `run.attach` refused every session that restarted after its idle timeout. > - Events after a terminal describe the past. Refusing the session cannot undo them, and it hides them. > - This pull request asks a different question: is anything live or waiting right now? Past events are recorded and no longer block. > - The benefit is that harmless events stop breaking session continuity, with no event list for Core and the Runner to keep in step. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for `run.attach requires a settled Codex provider session` and `prior_owner_terminal_state_indeterminate`. Related work: #18890 added one more accepted event to Core's list. This change replaces that list. #18991 handles late usage and is independent of this change. **What happened?** On a test deployment, 11 runs failed in 21 days when `run.attach` refused with `requires a settled Codex provider session with no pending events`. In all 11, the four pending events were two startup audit events, `session.resumed`, and a `codex_resume_usage_snapshot` diagnostic. The diagnostic is what the attach check refused. The retry then failed with `prior_owner_terminal_state_indeterminate` in 6 of them, which hid the first failure. **Expected behavior** A session with nothing live or waiting accepts the next run. If attachment fails, the run reports the real cause. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session. 2. Wait for the idle timeout so that the provider process stops. 3. Start the next run. Codex replays the usage for the settled turn on `thread/resume`. 4. The attach fails. `durable_backend_attaches_after_a_settled_resume_usage_snapshot` reproduces this on the unfixed code. **Paperclip version or commit** Observed on `8fd917aedf`. The same check is on `master` before this change. **Deployment mode** Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - **Runner attach:** `attach_run` refuses only for live or waiting state: no thread, a closed session, a session that needs reconciliation, an active or ambiguous turn, an active goal, a pending tool call, or an unanswered provider request. The error now names the reason. - **Pending events:** they no longer block attachment. They stay in the retained queue, and the Runner commits them under the prior authority before it rotates. `session.resumed` is still discarded because it is superseded. - **Runner drain:** `drain_completed_turn_tail_for_warm_attachment` no longer needs a list of known messages. It still fails on a message that names another turn, on `turn/started`, on a provider request, and on a process exit. - **Core:** `isSettledRemoteWarmControlState` accepts any authenticated event after the terminal unless it starts work (`turn.started`, `turn.accepted`, `run.attached`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created`), or a goal snapshot or update shows an active goal or work in progress. All identity, source and protocol checks are unchanged. - **Visibility:** the next run's log names the event types that were recorded after the previous run finished. - **Real cause:** a same-run retry after a failed attachment now fails with `prior_attach_failed: <original failure>`, instead of `prior_owner_terminal_state_indeterminate`. - **Docs:** added "Session settlement after a run" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 102 passed, 2 ignored. `--lib`: 478 passed, 2 ignored. `--test durable_recovery`: 3 passed. - **Failure before the fix:** `durable_backend_attaches_after_a_settled_resume_usage_snapshot` failed on the unfixed code with the same error as production. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence|recovers a blocked|automatically recovers only a proven settled local session|retained|warm"`: 476 passed, 680 skipped by the filter. - `pnpm exec vitest run src/services/native-runtime/native-session-resume.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - **Real data, read-only:** I decrypted the 11 failed runs' stored events on a test deployment and printed only event types, field names, and thread and turn IDs. All 11 show the same four events. - **Audit of live state:** I read the Runner's provider state and request tracking to build the attach check. It is a code audit. I have not proved that every kind of live work sets a flag. The Runner holds active turns, ambiguous starts, tool calls and provider requests in state today. - **Not covered:** the wiring that throws `prior_attach_failed` has unit coverage for its message only, not an end-to-end test. - **Review:** Greptile Apex scored 4/5, then 5/5 at the current head. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** no live run has used this change. ## Risks - **Fail-open for unknown live work:** an event type that starts work, but is not in the short list and sets no state flag, would now pass. This is the main tradeoff. The previous list failed closed, but it caused real outages. - **More events reach the journal after a terminal.** The usage replay diagnostic is now delivered rather than refused. These events are informational, and they are named in the next run's log. - **A failed attachment is not retried on the same run.** Some refusals could clear on their own, such as a turn that finishes, and some attach errors are not about live work, such as a failed provider restore. The run now reports the cause, and it needs another run to continue. On a test deployment, the next successful run after each failure used a new provider thread and a new execution workspace. A later change could add same-run retry. - **Rollout order:** deploy Core before the Runner. An older Core would refuse the new events that a new Runner commits after a terminal. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |