mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 22:56:58 +01:00
b2a7d322ed0bafda04a0b78ce14e4f25c8d5d170
2350
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
b2a7d322ed |
test: preserve runner commit stall acknowledgement invariants (#20383)
Assert that stalled event commits do not advance acknowledgement or committed state. Clarify the limits of historical failure attribution. Preserve the production implementation already merged in #20385. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
50df9a19b9 |
perf(inbox): read only the execution workspaces Mine rows reference (#20388)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Mine inbox is the first page a board user opens to see the tasks that need them > - The page loaded the company's full execution-workspace summary list and kept task rows hidden until that list arrived > - On a company with thousands of workspaces, that read checks each workspace row against the privacy predicate. It was the slowest request on the page > - The inbox needs workspace names only for the workspaces its rows reference, and only when the view shows workspace data > - This pull request adds an `ids` filter to the summary read, makes the inbox request only referenced workspaces, and stops rows from waiting for workspace data when the view does not show it > - The benefit is that Mine rows appear when the task data arrives, and the workspace read becomes a small bounded query ## Linked Issues or Issue Description **What existing behavior does this improve?** The Mine inbox (`/:companyPrefix/issues?view=mine`) shows its task rows late on companies with many execution workspaces. **Subsystem affected** Board UI inbox and the execution-workspace summary API. **Current behavior** The page calls `GET /api/companies/:id/execution-workspaces?summary=true` with no filter. On a large company this returned 8,758 rows (2.5 MB) and took 12.3 s on the server. The reveal gate included this read, so rows appeared at 13.4 s although `inbox/mine` arrived at 1.6 s. **Proposed behavior** The inbox requests summaries only for the workspaces its rows reference. Rows appear when the Mine task data arrives, unless the view shows workspace data. **Reason and benefit** The workspace read becomes a small bounded query, and it leaves the critical path for the default Mine view. **Breaking changes** None. The `ids` parameter is optional and additive. ## What Changed - `packages/shared`: add `executionWorkspaceSummaryIdsQuerySchema` (comma-separated UUIDs, 1–100 distinct values) and `EXECUTION_WORKSPACE_SUMMARY_IDS_MAX`. - `server`: `GET /companies/:companyId/execution-workspaces` accepts `ids`. Bad input returns 422. The service adds `inArray(execution_workspaces.id, ids)`. The company scope and the actor read condition still apply, so privacy does not change. - `ui/src/api/execution-workspaces.ts`: `listSummaries` accepts `ids`. New `listSummariesByIds` reads ids in chunks of 100 and concatenates the results. - `ui/src/pages/Inbox.tsx`: the workspace query is keyed on the sorted distinct `executionWorkspaceId`s of the rows the inbox holds (Mine, touched, context and remote search issues). It uses `keepPreviousData` and runs only when isolated workspaces are on and at least one id exists. - Reveal gate: rows wait for the workspace read only when workspace data changes what the rows show. This is workspace grouping, a visible Workspace column, or an active search (workspace names are searchable). Otherwise rows reveal without it. - Tests: route ids pass-through and 422 cases (non-uuid, empty, separators only, more than 100); service test for the id filter with company scope and the read condition; UI api chunking test; Inbox tests for "rows reveal while the workspace read is pending when no workspace column or grouping is active", rows waiting when the Workspace column is visible, and "shows the resolved isolated workspace name" with the id-filtered call. Follow-ups (not changed here, they still read the unfiltered list): - `ui/src/pages/LegacyInbox.tsx:772` - `ui/src/components/IssuesList.tsx:985` - `ui/src/components/LegacyIssuesList.tsx:888` - `ui/src/components/NewIssueDialog.tsx:570` (filtered by project and reuse, but not by ids) ## Verification Focused checks on this revision: - `cd server && npx vitest run src/__tests__/execution-workspaces-routes.test.ts src/__tests__/execution-workspaces-service.test.ts -t "summar|ids"`: 14 passed (embedded Postgres ran the service test). - `cd ui && npx vitest run src/api/execution-workspaces.test.ts src/pages/Inbox.test.tsx`: all passed. - `pnpm --filter @paperclipai/server --filter @paperclipai/ui --filter @paperclipai/shared typecheck`: passed. - `pnpm check:token-gates`: clean. - `pnpm verify` selected 2,481 Vitest files (the shared validator change reaches most suites). I stopped it at a 25-minute limit on a heavily loaded shared host. Before the limit, three server tests outside this change failed: `agent-commentary-skills` (skill mount), `agent-config-read-access` (24.8 s timeout) and `agent-directory-working-copies`. Required CI supplies the full coverage. - Review fix `fc3f82dae0`: when workspace data is visible, the reveal gate also waits while `keepPreviousData` supplies the previous id set's summaries. A new test switches Mine → Recent to a new workspace and checks the Recent row stays hidden until its workspace loads. It fails without the change. - Follow-up commits `bf34f84823` and `8fc743b5ca` add a 100-row Mine test with a counted workspace id scan. It fails with the old `= []` defaults and passes with the stable `NO_ISSUES` default. Performance (local, synthetic fixture): - Instance: isolated worktree instance, `authenticated` mode, signed in as a non-admin company member so the real workspace privacy predicate runs. Production UI bundle (`PAPERCLIP_UI_DEV_MIDDLEWARE=false`). Headless Chromium 1440×900, a new browser context per run (cold HTTP cache and cold query cache). macOS, Apple Silicon, with other local sessions running. - Fixture: 8,750 execution workspaces (each with a source task and a privacy-source key), 38,750 tasks, 150 Mine tasks; the Mine snapshot returns 100 rows that reference 100 workspaces. - Finish event: the first `[data-slot="task-row"]` is in the DOM. | Measurement | Before (`02bfe71a6b`) | After | | --- | --- | --- | | Cold time to rows, median of 7 | 1,209 ms (1,046–1,312) | 702 ms (567–1,209) | | Workspace summary read, server wait (time to first byte), median of 7 | 496 ms unfiltered, 2.0 MB | 18 ms with 100 ids, 22 KB | The difference of medians is 507 ms. This is close to the unfiltered read's wait on this fixture (about 550–590 ms in the browser runs). The runs were sequential batches, not alternating pairs. Limits: this fixture does not reproduce the 12.3 s server time seen on the development instance. Here the unfiltered read cost 0.5–1.7 s and grew with the company's task count. The browser gain on the original instance should be close to that instance's unfiltered read time, but I did not measure it there. ## Risks - Low risk. The `ids` filter only narrows the existing query. It does not change the read condition or the company scope. - A row whose workspace is not readable by the actor shows no workspace name, as before. - When the Workspace column or grouping is visible, rows still wait for the workspace read. That read is now small. - When the id set changes, the query refetches all chunks. Before a view first reveals, rows wait for the new summaries if workspace data is visible. After that, the previous names stay visible while a refresh loads. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, headless browser). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b09f17fbfd |
fix(agents): repair queued setup through task connection cards (#20361)
## Thinking Path > - Paperclip manages AI agents doing work for people. > - Agent verification must finish before an accepted task can start. > - A missing AI account default can prevent verification before a provider starts. > - Lifecycle policy treated that owned configuration refusal as a temporary failure and retried each minute. > - The accepted turn remained Queued without explaining the prerequisite. > - Park account configuration failures and reuse the task's existing inline connection card. > - Successful repair rechecks verification and releases the original queued turn under its original authority. ## Linked Issues or Issue Description **What happened?** An OpenCode agent selected the responsible user's OpenRouter default, but that default did not exist. Verification retried unchanged configuration indefinitely and its accepted task stayed Queued. **Expected behavior:** Preserve accepted work, explain the setup prerequisite, and let the appropriate person repair it using the existing task connection UI. Completion must verify the agent before starting the same queued turn. Genuine pause, approval and access gates remain effective. **Reproduction:** Create an OpenCode Runner agent with a responsible-user OpenRouter binding and no personal default; assign a task while verification is pending. The base retries verification and shows an unexplained queue. Related: #18702 handles pending task connection recovery after account repair. This PR handles the earlier agent-verification boundary. #20330 addresses a separate Codex governed-wait timeout and is already merged. ## What Changed - Share owned account-selection failure evidence through a pure library contract. Lifecycle policy parks those configuration failures with fixed, redacted guidance; temporary infrastructure errors retain their retries. - Batch setup-state reads for unstarted queued turns. Distinguish manual repair from automatic setup retries and preserve Stop, started/history state and lifecycle holds. - Reuse the existing `connection_intent` card, account selection and completion APIs. Its controller-only `agentSetup` marker accepts only never-started legacy turns; runtime tokens still require a running run. - Address the card to the lifecycle verifier's account owner, preserving the task's execution principal and private-task visibility. Attach missing cards on admission, verification, startup and full recovery, including retained retries and continuations. Repeated delivery and publication remain bounded. A new queued turn supersedes the setup card of a stopped turn; generic connection cards retain cross-run reuse. - Recheck lifecycle verification after usable-account completion, then release the existing turn through the normal readiness path. Preserve decline, Stop, reassignment and changed setup-owner gates. Fence an in-flight verification failure from overwriting a repair; add no provider wake or recurring worker. - Park owned account refusals during resume as well as initial verification. Bind setup cards to the lifecycle operation, settle obsolete deliveries, and keep explicit Retry guidance accurate. - Reuse the router’s existing lifecycle pin for setup repair without rerouting the task, advancing the pool cursor, or replacing its saved binding. Legacy direct-account cards remain readable. - Hide the incorrect Resume banner during setup, retaining genuine manual/budget pause controls. Document the execution and retry contract. ## Verification - Red: eight original behavior assertions failed on the base, including the real selector's retry decision and the real batched queue projection. Six owned selection cases failed classification. Temporary-error presentation, different task/verifier users, retained retry/continuation recovery and the setup Resume banner also have failing-before/passing-after regressions. - Green: 430 affected backend, lifecycle, shared-contract and UI tests passed before the final review corrections. Subsequent focused runs cover one inline card, real account selection/completion, duplicate delivery/publication, verification before readiness, unchanged run authority, private tasks, no extra provider wake, decline/Stop/reassignment, company isolation, invalid runtime tokens, transient retries and redacted diagnostics. - Final setup/card and generic connection-service suites: 49 tests passed. The stopped-turn case failed before the correction, then passed with real account completion, the stopped source remaining cancelled, and the replacement source remaining queued. - After rebase, all 129 focused assertions passed. The local queue suite’s 88 execution assertions passed repeatedly, but its database-fixture teardown exceeded both 30-second and 120-second hooks. An isolated queue case completed, and the setup/connection suites fully passed. The original queue test and cleanup remain unchanged; broad CI supplies the full-suite result. - Server/UI typechecks, module boundaries, token gates and diff checks passed. `pnpm verify --explain` requested broad CI because the execution-contract document changed; no redundant full local suite was substituted. - Real source acceptance: macOS arm64, Node 26.4.0, isolated embedded PostgreSQL and local board identity. Created an OpenCode/OpenRouter agent and task through the API with no default. Admission retained one unstarted turn and produced the existing inline Connect OpenRouter card. The browser opened its credential form and closed it without entering credentials. The same card/run persisted after server restart and reload, with the setup label and no incorrect Resume. The browser also reproduced Stop followed by a new queued turn retaining the stale card on the prior head. On `12f609c98372fd1e50644028db75cfcc8deb210d`, startup superseded the stale card and retained the same replacement turn, with the original turn still cancelled and no provider execution. - The earlier CI case-search failure also reproduces on exact base `0e3fdebea8257bdff11573081c76b306f12db230` without this PR. Its repair is now in master via #20371; this branch is rebased onto it and contains no API-search changes. - Staging's actual Runtime test returned “Connect an account and choose your personal default” before provider startup. The original turn later started independently on deployed code, then hit a separate workspace-export failure. Automatic file recovery subsequently marked that run successful; its task remains Blocked and the trial’s useful output is unverified. This PR is not deployed; provider authentication through its card and downstream publication remain unverified. No account, credential, default, permission or binding was changed during diagnosis. - Previous revision review: Greptile 5/5 on `12f609c98372fd1e50644028db75cfcc8deb210d`, with all four inline findings addressed and resolved. [Full CI passed](https://github.com/paperclipai/paperclip/actions/runs/42713494508) on that revision: 59 successful checks, one neutral check and three intentional skips, with no pending or failed checks. - Latest revision `976bcae993805a8a2222395bbff9394f6791b424`: resume, router and Retry-copy regressions failed before repair. Green: 92 lifecycle/setup/shared tests, 83 setup/generic-connection/router tests, and 137 UI/shared tests. Server/UI typechecks, module boundaries and token gates passed. [Full CI passed](https://github.com/paperclipai/paperclip/actions/runs/42720731137) on this exact revision: 59 successful checks and three intentional skips, with no pending or failed checks. Greptile reviewed this head at 5/5; all six review findings are addressed and resolved. Real Stop/retry acceptance retained the same replacement turn and current-operation card through two actual server restarts, with no provider execution. ## Risks - Known account configuration failures now require repair or explicit Retry. Temporary failures keep automatic retries; accepted work and authority remain intact. - The projection adds one company-scoped batch read only for unstarted queued turns. Card attachment examines at most 100 active candidates and adds no timer, worker, schema or migration. - The optional stored setup-card marker requires an aware reader during rollback. No run/session/checkpoint identity format changes. Setup presentation grants no access and cannot bypass verification or other gates. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use, code editing and test execution. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7e93ccb4b3 |
perf: fetch recent task summaries in one request (#20389)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI sidebar shows the person's recent tasks on every page, including the Mine inbox > - The sidebar refreshes those rows through `GET /companies/:id/issues/summaries`, which accepted at most five IDs > - `useRecentTasks` sent the chunks one after another, so a 25-task history cost five sequential requests > - On staging each request took 3–6 s, so the recent-task rows settled about 20 s after load and every request competed with the inbox rows > - This pull request raises the per-request cap to 50 and fetches any remaining chunks in parallel > - The benefit is one summaries request for a typical history instead of a serial chain ## Linked Issues or Issue Description **What existing behavior does this improve?** The refresh of the sidebar's recent tasks on page load. **Subsystem affected** Board UI (`ui/src/hooks/useRecentTasks.ts`) and the `/issues/summaries` route bound (`packages/shared`, `server/src/routes/issues.ts`). **Current behavior** A history of N recent tasks sends ceil(N/5) summaries requests in series. Measured on staging on 2026-11-09 for a cold Mine load: five requests of five IDs, started at 1.0 s, 4.3 s, 8.1 s, 12.3 s and 18.4 s, each taking 3.3–6.1 s. **Proposed behavior** Up to 50 recent tasks are one request. Longer histories send their chunks at the same time. **Reason and benefit** The staging server is saturated, so each extra serial round trip delays the sidebar and the inbox rows. One request removes four round trips from a typical load. ## What Changed - `ISSUE_SUMMARIES_MAX_IDS` is now 50 (was 5). The shared validator, the route's 400 message, the service guard and the OpenAPI description use the new bound. - `useRecentTasks` builds all chunks first and fetches them at most three at a time (`RECENT_TASK_SUMMARY_CONCURRENCY`). The recent list has no size cap, so this keeps a long history from becoming a burst. The cache key, omission handling and pruning are unchanged. A failed chunk still fails the batch, so nothing is pruned on a partial failure. - Tests: the route test accepts exactly `MAX` IDs and rejects `MAX + 1`; a hook test checks that at most three chunks are in flight; the API test uses the constant; `useRecentTasks` tests cover a 25-task history in one request and a 103-task history with three chunks in flight together, with and without a failed chunk. - New browser check `tests/perf/collections/recent-task-summaries.spec.ts`: a Mine load with a 12-task history sends summaries requests that each carry all 12 IDs, and the server answers 200 with all 12 tasks. ## Verification - `pnpm verify` (selected Vitest files) passes. - `server`: `npx vitest run src/__tests__/issue-summaries-routes.test.ts src/__tests__/issue-display-status.test.ts` passes (embedded PostgreSQL). - `ui`: `npx vitest run src/hooks/useRecentTasks.test.tsx src/api/issue-summaries.test.ts src/components/task-chat/TaskChatBlockerLinks.test.tsx src/components/CoordinatorSidebar.test.tsx src/components/SidebarRecentTasks.test.tsx` passes. - Browser check against a disposable local instance: `PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/collections/playwright.config.ts recent-task-summaries.spec.ts`. It passes on this branch. With `master`'s hook and constant it fails: three requests of 5, 5 and 2 IDs. ## Risks - Low risk. The route still validates UUIDs, deduplicates, and applies the same read authorization per row. Blocker attention is computed with set-based queries, so 50 IDs is the same number of queries as 5. - During a rolling deploy, a new UI against an old server would get a 400 for a history above five tasks. The server serves the UI bundle, so both change together on a normal deploy. The sidebar keeps its stored snapshot when the refresh fails. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser). 1M-token context window. Extended thinking was off. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
5858cef8f9 |
perf(runner): poll the provider right after an ACK releases it (#20374)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner (`paperclip-runnerd`) sends provider events to the controller over one authenticated socket > - Its main loop reads one control frame, then polls the provider only after a read finds no frame > - After the controller ACKs the last sent event, the next read waits the full 250 ms `RUNTIME_READ_TIMEOUT` before the Runner polls the provider again > - A Codex turn start needs one or two such rounds before the Runner reads `turn/started`, so every Codex turn waits 250 ms or 500 ms for no reason > - This pull request reads only the frames that are already queued after a consumed control frame, and then polls at once, when this connection produced the unacknowledged output > - The benefit is 250 ms less time to `turn/started` for each ACK round, and faster event delivery after each ACK ## Linked Issues or Issue Description This is an enhancement. The fields follow `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** Codex turns wait for the Runner read timeout before the Runner sends `turn/started`. **Subsystem affected** The Runner (`paperclip-runnerd`) durable loop in `packages/paperclip-runner/runner/crates/runner-core/src/durable/`, for the local Codex provider. **Current behavior** On a hosted test deployment (48 h, 114 Codex turns), `runner.transport.started_event` for Codex has two modes: | Codex turn | n | `started_event` p50 | | --- | --- | --- | | no provider cold start in the trace | 57 | 468 ms (all < 700 ms) | | provider cold start in the trace | 57 | 918 ms (51 of 57 ≥ 700 ms) | For Claude (ACPX) turns, the p50 is 90 ms. ACPX returns `turn.started` with the command result. Codex reads `turn/started` from the provider in a later poll. **Proposed behavior** The Runner polls the provider as soon as the controller ACK releases the poll gate. The time to `turn/started` then follows the controller round trip, not the read timeout. **Reason and benefit** - Real Codex (`codex app-server` 0.157, local) sends `turn/started` 2–16 ms after its `turn/start` response. The provider is not the cause. - `runner.transport.started_event` starts when the `turn.start` command result arrives. It ends when the controller commits `turn.started`. In this time the Runner sends `turn.accepted`, waits for its ACK, waits 250 ms, polls, and sends `turn.started`. - On the first turn after a provider launch, the first poll returns only the launch diagnostic that the executor retained. A second ACK round and a second 250 ms wait come before the poll that reads `turn/started`. This is the difference between the two modes above (about 450 ms: 250 ms wait and one event round). - So 250 ms (warm turn) or 500 ms (first turn) of each Codex `started_event` is the read timeout. The rest is one or two controller event rounds (about 200 ms each on that deployment), less than 20 ms of Codex time, and the 10 ms pump interval of the TypeScript transport. - The expected benefit is 250 ms less per warm Codex turn start and 500 ms less for the first turn after a provider launch, plus faster Codex event delivery after each ACK. **Breaking changes** None. No protocol, schema, or controller change. ## What Changed - `AuthenticatedTransport::receive_queued_json` reads with a 1 ms timeout and then restores `RUNTIME_READ_TIMEOUT`. - New `CommandExecutor::provider_poll_is_live` hook (default `false`). The local Codex executor returns `true` only when it owns a running provider with no open startup attempt, so a poll cannot restore or launch a provider. The native executor forwards it for the local Codex facade only. - In the steady-state loop, after a consumed control frame, the Runner uses `receive_queued_json` when it has sent output that this connection produced (a command or a provider poll) and the executor reports a live provider. If no frame is queued, it polls the provider at once. - When a provider poll returns no output, or when no frame was consumed, the Runner uses the 250 ms read as before. A 1 ms read follows only a consumed frame, so an idle loop does not spin. - The replay boundary is the highest source sequence before the welcome's pending commands run. A fresh command delivered in the welcome counts as this connection's output. Replayed commands only resend their stored results. - Replayed output from an earlier connection or process, ACPX, and the other providers keep the 250 ms read. The controller can still send its next command before a poll restores a provider. (Without the live-provider gate, six ACPX source-recovery tests in `runnerd-acpx-source-reconciliation.test.ts` failed: the fast poll restored the ACPX sidecar before the next recovery command.) - Control frames that are already queued behind the ACK are still read and executed before the provider poll. The ACK gate in `poll_executor_events_after_controller_ack` is unchanged. ## Verification - `cargo test -p paperclip-runner-core --lib durable::` (121 passed). Five new tests in `src/durable/tests/ack_release_poll.rs`, included by the transport test module like `hosted_retention.rs`: - `acknowledged_output_polls_the_provider_without_waiting_out_the_read_timeout`: the provider poll follows the ACK read in less than `RUNTIME_READ_TIMEOUT`. Without the change it fails (the poll waited 280–356 ms). - `control_queued_behind_an_ack_runs_before_the_next_provider_poll`: a command sent in the same write as the ACK executes before the provider poll. - `welcome_command_output_polls_the_provider_without_waiting_out_the_read_timeout`: the output of a fresh command in the welcome also releases the poll at once. With the replay boundary taken after the welcome commands, it fails. - `replayed_output_ack_keeps_the_read_timeout_before_the_provider_poll`: after a reconnect, a command sent 50 ms after the ACK of replayed output executes before the provider poll. Without the replay condition it fails. - `restorable_provider_ack_keeps_the_read_timeout_before_the_provider_poll`: when the executor does not report a live provider, a command sent 50 ms after the ACK executes before the provider poll. - `npx vitest run src/live/runnerd-` in `packages/paperclip-runner` with the new debug `paperclip-runnerd`: run with `--no-file-parallelism`. Before the welcome-boundary fix: 576 passed, 3 skipped. After it: 575 passed, 3 skipped, and 1 failure once (`stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`, a SIGKILL timing test) while `cargo test` ran at the same time. `runnerd-codex-transport.test.ts` then passed 293 of 293 three times with this build, and three times with the base build. - Local before and after measurement with the real Runner, the TypeScript transport, and the fake Codex app-server (8 turns, 2 runs each, debug build): | span | before p50 | after p50 | | --- | --- | --- | | `runner.transport.started_event` | 377 ms (first turn 682–695 ms) | 119–122 ms (first turn 117–119 ms) | | `runner.transport.turn_start` | 462–471 ms (first turn 775–780 ms) | 204–212 ms (first turn 247–251 ms) | - The saving on the hosted deployment is not measured. From the analysis above, the expected saving is about 250 ms for a warm Codex turn and about 500 ms for the first turn after a provider launch. ## Risks - Only local Codex gets the fast path. ACPX streaming keeps the old 250 ms wait after each ACK. A later change can opt ACPX in when its poll can prove that it does not restore the sidecar. - More provider polls during Codex streaming: a poll can now follow each ACK at once instead of after 250 ms. Batches can become smaller, so the provider state save for each batch can happen more often. Each event already has its own durable save. - A command that the controller sends more than 1 ms after its ACK can now arrive after a provider poll. This was already possible after 250 ms. The change applies only when this connection produced the output, so the provider is already active. - If the Runner cannot restore the 250 ms read timeout, it treats the socket as failed and reconnects. It does not continue with a 1 ms timeout. - No protocol, schema, or controller change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, Honeycomb queries) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c985541a57 |
perf(db): index retry-exhaustion receipts for the attention scan (#20386)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The attention feed shows runs that failed after automatic retries stopped > - To find these runs, the feed reads every `heartbeat_run_events` row of the company and filters the rows by a message prefix > - Run output makes this table very large, so each feed request does a large scan > - On a hosted test database, this one statement used the most total database time: 923 calls at a mean of 5 s (4,709 s in total) > - Request authentication waits behind this background work, so all users feel the load > - This pull request adds a small partial index that holds only the retry-exhaustion receipts > - The benefit is that the same query reads a few index entries instead of all company events ## Linked Issues or Issue Description **What existing behavior does this improve?** The attention feed query that finds retry-exhausted runs (`listAttentionExhaustedRuns`). **Subsystem affected** Server attention feed and the `heartbeat_run_events` table schema. **Current behavior** The query reads all company rows in `heartbeat_run_events`, filters them by `event_type = 'lifecycle'` and the `Bounded retry exhausted` message prefix, and then joins the few matches to failed or timed-out runs. The cost grows with all run output. It does not grow with the number of exhausted runs. **Proposed behavior** A partial index holds only the matching receipts. The query plan reads the index and joins a few rows. The query text and the result do not change. **Reason and benefit** The scan was the most expensive statement by total time on a loaded hosted database. A local reproduction shows a 54x faster query and about 116x fewer buffer reads. **Breaking changes** None. The API, the feed rows, and their order do not change. ## What Changed - Add the partial index `heartbeat_run_events_retry_exhausted_idx` on `(company_id, run_id, id DESC)`. The index covers only rows where `event_type = 'lifecycle'` and `message like 'Bounded retry exhausted%'`. - Add migration `0454`. It uses `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. The migration runner runs it as a top-level statement, the same as migrations 0432, 0436 and 0450. - Add a regression test. It pins the row order (newest run first, then the newest receipt on a tie) and the latest receipt per run. It also checks that the query plan can use the new index. If the query predicate changes so that the index no longer applies, the test fails. - Add `scripts/benchmark-exhausted-run-attention.ts`. It measures the same query with and without the index, and the time to build the index. - Document the index in `doc/DATABASE.md`. ## Verification Cause and thesis. The scan cost comes from reading all company events, not from the few exhausted runs. Thesis: an index that holds only the receipts makes the query cost depend on the number of receipts, and keeps the result identical. The benchmark below confirms the thesis. Rejected alternative without a migration. I first tried a LATERAL lookup that starts from the failed and timed-out runs. On the same fixture it was about 2x slower (median 379 ms against 194 ms). It reads the full event history of each failed run with random I/O, and that costs more than one sequential scan. Local SQL benchmark (the SQL that the service generates; same SQL with and without the index): ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-exhausted-run-attention.ts --runs=30000 --events-per-run=100 --samples=9 ``` Fixture (synthetic): one company with 30,000 runs (3,819 failed or timed out) and 3.0 M events, plus a decoy company with 0.5 M events. The table is 1.4 GB. About 5% of events are `lifecycle` events. Events of 20 concurrent runs are interleaved. 1 in 23 runs has a receipt, some runs have two receipts, and output decoys contain the prefix. Two agents are terminated. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted database is Neon with different compute, so the absolute times are local SQL measurements only. The script runs two cycles (without index, then with index; then with index, then without index) to expose cache bias. Each measurement has 2 warmups and 9 samples. | Query | Without index | With index | Samples and spread | Output check | | --- | --- | --- | --- | --- | | `listAttentionExhaustedRuns` | median 157 ms (96–1,271 ms) | median 3.3 ms (2.2–5.7 ms) | 18 each | 133 rows, positional rows equal in all 4 measurements | | Buffers (EXPLAIN ANALYZE) | 127,743 blocks | 5,580 blocks | | plan uses the new index (asserted) | Speedup: 48x (median), on head `ae6037b2a0` (the same SQL and index as the final head). The machine was under heavy load from other work, so the spread without the index is wide. An earlier run on the first head gave 248 ms against 4.6 ms (54x). Index build (`CREATE INDEX CONCURRENTLY` from the migration file): 1.7 s and 2.5 s for the 1.4 GB table. Index size: 152 KiB. Review fix: the query now writes `event_type = 'lifecycle'` as a SQL literal, not a bound parameter. With prepared statements, PostgreSQL can reuse a generic plan. A generic plan cannot prove that a bound parameter matches the index predicate, so it would not use the index. Tests: - `cd server && npx vitest run src/__tests__/attention-service.test.ts`: 44 passed. The new test checks a fresh plan and a forced generic plan (`plan_cache_mode = force_generic_plan` with `PREPARE`). It failed with the bound `event_type` parameter and with a changed message prefix (confirmed red), and it passes with the fix. Locally, the file's `afterAll` database cleanup hook timed out on a heavily loaded machine (load average about 28). The timeout also occurs when the new test is excluded. - `pnpm --filter @paperclipai/db check:migrations`: passed. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck`: passed. - `pnpm verify`: see the CI results for this head. ## Risks - Migration time. A concurrent index build reads the full `heartbeat_run_events` table and does not block writes. Locally, the build took 1.2–2.5 s for 1.4 GB. The time grows linearly with the table size. I did not time the build on a production-size clone. On a database with a 5 s full company scan, I estimate the build at tens of seconds, which is in the 1–2 minute budget. This estimate is not proven. - If the build stops, PostgreSQL can leave an invalid index. The migration runner repairs invalid concurrent indexes when it runs again, and `IF NOT EXISTS` makes a rerun safe. - The index predicate must stay textually equal to the query predicate. If someone changes the message prefix or binds a predicate value as a parameter, the planner cannot use the index. The new test catches both cases, for fresh and generic plans. - Each new receipt writes one more small index entry. Receipts are rare, so the write cost is very small. - The hosted improvement is not verified until deployment. After deployment, compare a new query-statistics window for this statement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local PostgreSQL). Standard context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
199670ba99 |
fix(accounting): prevent usage receipt stalls during streamed run updates (#20385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner commits usage receipts before it acknowledges retained events. > - Usage receipts and streamed run logs update the same run row. > - A repeatable-read snapshot can become stale before the receipt writer locks that row. > - Continuous log writes can exhaust all accounting retries and stall the retained event. > - This pull request uses the existing run lock with READ COMMITTED for standalone receipt writes. > - Other accounting operations keep their consistent snapshot. Stall diagnostics retain safe nested error codes. ## Linked Issues or Issue Description Related: #20235. **What happened?** A hosted native run stopped on a retained usage event. Its diagnostic reported `Error at source seq N`. The saved outbox and accounting records locate the failing delivery boundary. Traces show repeated accounting transactions while streamed events still updated the run row. The historical inner exception is unavailable. A controlled PostgreSQL reproduction proves a matching defect: a run-log append after company validation makes the receipt writer's repeatable-read snapshot stale. Repeated appends exhaust all four retries with SQLSTATE `40001`. This is a demonstrated failure mechanism; attribution of the historical incident remains an inference. **Expected behavior** Commit the receipt under its existing run-row lock despite concurrent run-log writes. Observe concurrent finalization and preserve late-receipt debt. Keep duplicate receipts idempotent and reject conflicting or cross-company receipts. **Steps to reproduce** Run `server/src/services/usage-receipts-read.test.ts`. The concurrency regression uses real PostgreSQL and the production run-log append function. It commits a log append after company validation and before the receipt locks the run. Before the repair, the receipt rejects with a wrapped `40001` after four attempts. After the repair, it commits once. The finalization variant verifies that a newly settled run receives late-receipt debt without replacing settled state. ## What Changed - Allow aggregate-lock-protected accounting writers to select READ COMMITTED. Keep repeatable read as the default. - Use READ COMMITTED only for standalone usage receipt writes. Keep the existing run lock, immutable receipt journal, replay checks and transactional budget-work publication. - Add PostgreSQL regressions for concurrent log appends, concurrent finalization, duplicate delivery and the unchanged default snapshot. - Read the outer event-commit error and at most three nested causes for an existing safe error code. Exclude exception messages and SQL. - Document receipt isolation and bounded stall diagnostics. ## Verification - Red: the production receipt writer exhausted four retries with PostgreSQL `40001` under controlled concurrent run-log appends. - Red: wrapped database and transport stall diagnostics returned `Error` instead of `40001` and `ECONNRESET`. - Green: Runner control-plane tests passed all 207 cases. All 68 server assertions passed across receipt reads, native usage delivery and cost accounting. Cost-accounting teardown exceeded its 30-second limit twice on this loaded host, after all 28 assertions passed. The workspace project retained its configured 30-second hook limit despite the CLI override. This is a local suite cleanup failure, not a passing suite; the same revision passed the complete server suites in CI. - Server layered TypeScript checking passed, including source and test layers. Runner TypeScript checking passed. Server checks cover `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`; the unchanged Runner checks cover its parent. - Local environment: macOS arm64, Node 26.4.0, disposable embedded PostgreSQL. Native usage-delivery tests use the real Rust receiver. - CI at `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`: [build](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568428), [typecheck](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568367), native Runner integration, browser shards and all nine serialized server groups passed. All 12 general server shards passed in the [full run](https://github.com/paperclipai/paperclip/actions/runs/42714760702). Final rollup: 59 successful checks/statuses, three intentionally skipped checks, no pending or failed checks. - [Greptile reviewed this revision at 5/5 with no actionable findings](https://github.com/paperclipai/paperclip/pull/20385#issuecomment-6753163397). - Hosted evidence came from the task, run, saved sandbox outbox, accounting receipts and traces. The run started fresh and did not cross a deployment. No original agent turn was replayed. - The hosted repair has not been deployed. The original task has not been resumed. A fixed-revision remote-provider user journey remains unverified. ## Risks - READ COMMITTED changes snapshot timing for standalone receipt writes. The run-row lock protects all mutable receipt/accounting reads and serializes competing receipts and finalization. Other accounting operations retain repeatable read. - Receipt identity, replay conflict checks, company scoping, budget publication, retry bounds and acknowledgement rules remain intact. - No persisted format, provider identity or database schema changes. Existing saved state needs no migration. - Diagnostics inspect at most four objects and retain only the existing identifier format, with an 80-character code limit. - The reproduced race matches the incident boundary and timing, but the original inner exception was not recorded. Rollout and live acceptance remain necessary before incident resolution. ## Model Used - OpenAI Codex, GPT-6 family, with code and shell tools. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
09241442ea |
fix(server): let a native reviewer approve its review stage (#20373)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution policies can require an independent review stage before a task is done > - A native (Runner) agent that is the active reviewer had no way to record its decision > - `PATCH status: "done"` is refused for a native run that owns the task (#18411), and `paperclip_finish` accepts the report but leaves the stage pending > - The reviewer then searches the API until a deadline, and the task stays in review with nobody able to decide it > - This pull request records the final-stage approval on `PATCH` without releasing the run's lock, and lets the run's own finish report set the final status > - The benefit is that an agent reviewer can actually approve its review stage, in one short turn ## Linked Issues or Issue Description Refs #18411 (added the native status guard). Related: #20264 (explicit stage-decision operation, still open; this change is narrower and compatible with it). **What happened?** A native reviewer agent was woken as the active `review` participant. It read the deliverable and wrote its review in about 100 seconds. It then called `paperclip_finish` with `done`. The tool answered "Completion report accepted; the task's configured review stage is still pending". It then tried `PATCH /api/issues/{id}` with `status: "done"` and a comment. The route refused it with `native_run_status_reported_by_finish`. The agent searched the API for 2 more minutes and found no other path. The stage stayed pending. When the run ends with `done`, the status arbiter sees the pending stage and keeps the task in review (`governed_gate_pending`), so the stage can never resolve. **Expected behavior** The active review/approval participant can approve its stage from a native run. The task reaches `done` after the approval. The implementer still cannot approve. A run that is not the active participant still cannot change the task status. **Steps to reproduce** 1. Create a task with an execution policy that has one `review` stage with a native agent participant. 2. Move the task to `in_review` so the stage is pending and the reviewer agent wakes. 3. In the reviewer run, call `PATCH /api/issues/{id}` with `{ "status": "done", "comment": "Approved: ..." }`. 4. Observe `409 native_run_status_reported_by_finish`. Call `paperclip_finish` with `done` and observe that the stage stays pending. ## What Changed - `PATCH /api/issues/:id`: when the requester is the native run that holds the task lock and is the active participant, a `status: "done"` request on the final stage now records the approval decision (`executionState` completed, decision row, activity). The route removes `status` from the write, so the task stays `in_review` and the run keeps its execution lock. The run's `paperclip_finish` report then sets `done` through the normal status arbiter. - An intermediate-stage approval (which would hand the task to the next participant) and every other status change from such a run keep the existing `native_run_status_reported_by_finish` refusal. - `paperclip_finish` from the active participant (`done` or `needs_review`) while its stage is still pending is now rejected with the exact decision path, instead of being accepted with "stage still pending". - The native finalizer now treats the final-stage approval that the run itself recorded (read from the `issue_execution_decisions` row, bound to this run and agent) as the completion authority for its clean `done` report. Before this, a reviewer that cited free-text evidence had its report graded as incomplete, and the task was reopened as `in_progress` with a new wake for the reviewer. Failed verification, blocking remaining work, attention requests, other dispositions, and a still-pending stage keep the old outcome. - The reviewer/approver wake text names the decision path (`PATCH` with `status` and a required decision comment) and says a completion report does not record the decision. A Runner reviewer that needs changes is told to report them with `paperclip_block`, because a changes request from the run that owns the task is a reassignment that stops that run. - The locked review-policy recheck treats the deferred approval as a verdict, so a policy changed to `human_only` before the locked write still refuses it. ## Verification - `pnpm exec vitest run server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts server/src/__tests__/issue-execution-policy-routes.test.ts server/src/__tests__/issue-comment-reopen-routes.test.ts server/src/__tests__/issue-execution-policy.test.ts packages/adapter-utils/src/server-utils.test.ts`: 5 files, 595 tests pass. - `pnpm exec vitest run server/src/services/native-runtime/native-completion-feedback.test.ts`: 29 tests pass (embedded PostgreSQL), including the decision-row binding for `executionStageApprovedByRun`. - `pnpm exec vitest run server/src/services/native-runtime/status-arbiter.test.ts server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts server/src/services/native-runtime/native-stale-owner-completion.test.ts`: 72 tests pass. - New route tests fail without the route change (409 instead of 200/422) and pass with it. The new feedback test fails without the feedback change ("Completion report accepted" instead of a rejection). - `npx tsc --noEmit -p server` and `-p packages/adapter-utils`: pass. - A route test changes the review policy to `human_only` between the first check and the locked write; the deferred approval is refused (403). - Live full-stack chat eval `task-coordination.independent-review` (Runner, OpenCode) with this change: passed in GitHub Actions run 42705964209 (eval branch with the first two commits cherry-picked). The reviewer run recorded its approval 24 s after its review brief was released, reported `done` 14 s later, and the finalizer set the task to `done` with reason `execution_stage_approved`; total reviewer run 73 s. The earlier run 42702224713 exposed the finalizer evidence issue that the second commit fixes. ## Risks - Low to medium. The change is limited to one request shape: a native run, holding the lock, that is the active participant of a pending final stage, sending `status: "done"`. All other requests keep the old guard. - Between the approval and the finish report, the task shows `in_review` with a completed `executionState`. If the run fails before it reports, the existing native recovery owns the task, as for any other run. - Previous-release story: no persisted shape changes. A reviewer run that started before the deploy could not have recorded a deferred approval (the old route refused it), so the new finalizer input is false for it and the old outcome applies. - Intermediate-stage approvals and changes requests from native runs are still refused by the route. A Runner reviewer reports required changes with `paperclip_block`, which keeps the stage pending for a person. A self-service changes request needs the deferred handoff semantics in #20264. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
facd409073 |
perf(workspaces): index legacy workspace companion lookups (#20372)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration must retain prepared sources and recovery holds. > - Companion discovery reads saved workspace references from run profiles. > - The selector can scan large saved profiles even when it returns few rows. > - This pull request adds one company and workspace expression index. > - Companion discovery uses the same expression as existing migration lookups. > - The result preserves each source check and reduces history scan work. ## Linked Issues or Issue Description **What happened?** The companion selector used a JSON path expression with no matching index. A read-only plan on about 22,000 run rows showed a sequential scan over a table with about 1.3 GB of table and TOAST storage. A repeated inspection reached its statement timeout. Existing migration lookups use a different JSON operator form, so one expression index must match both callers. **Expected behavior** Use a company-scoped indexed lookup. Keep prepared and finalized references, the exact text comparison, native issue references, and run order. Keep all source and recovery validation. **Steps to reproduce** 1. Retain many run profiles with saved workspace references and wide JSON fields. 2. Inspect a workspace with few matching companion runs. 3. Explain the company and saved-workspace selector. Without the new index, it can scan run history and read unrelated JSON values. **Paperclip version or commit** Baseline: `0e3fdebea8257bdff11573081c76b306f12db230`. **Deployment mode** Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1. The observed server plan used PostgreSQL 17.11. Related: [batched companion discovery in #19735](https://github.com/paperclipai/paperclip/pull/19735). The existing batching remains in place. The duplicate search found no open workspace index change. This branch also carries the isolated API route-summary correction shared with [#20370](https://github.com/paperclipai/paperclip/pull/20370). ## What Changed - Add one concurrent index on company ID and the saved native workspace ID expression. The index has no state, runtime, or status predicate. - Use nested JSON operators in companion discovery. Existing migration lookups already use this expression. Keep both accepted states and the same ordering. - Register the migration on the existing top-level migration path. Retain its advisory lock, interrupted index repair, and completion journal behavior. - Add a real PostgreSQL migration and ordered-result test. Update the existing companion query counter for the equivalent SQL form. - Generate the schema snapshot and retain the five newest snapshots. - Describe the task case-list route as “Find the cases for a task.” This keeps the route on the first discovery search page. Search logic and test assertions are unchanged. ## Verification - `pnpm --filter @paperclipai/db exec vitest run src/heartbeat-workspace-companion-index.test.ts --maxWorkers=1`: **1 passed**. It builds the concurrent index through the canonical migrator, checks index validity, preserves prepared/finalized and cross-issue/null-issue results, excludes foreign and malformed references, and checks one journal entry after a second migration call. - `pnpm --filter @paperclipai/server exec vitest run src/services/remote-workspace-online-source.test.ts -t 'discovers all Git companions' --maxWorkers=1`: **1 passed; 86 not selected**. This uses the real source owner and retains finalized recovery holds, company isolation, and a fresh obligation between inspections. - `pnpm --filter @paperclipai/db typecheck`, migration numbering/safety checks, and `git diff --check`: **passed**. - One small local SQL comparison used the captured baseline selector, the changed selector, 3,012 synthetic rows, about 37.8 MB of table storage, and the new expression index. Ordered results were identical. One client timing changed from **8.70 ms to 0.86 ms**. This is one local warm-cache comparison, not a server latency claim. Hosted improvement is **unverified**. - Initial setup attempts failed before the affected test ran: registry access, local tsx IPC, and a missing SDK build. The canonical Node 24.22.2/pnpm 9.15.4 setup and SDK prerequisite resolved these setup failures. The local SQL timing client used Node 26.4.0; the focused application tests used Node 24.22.2. - [The first CI run](https://github.com/paperclipai/paperclip/actions/runs/42705129724/job/129460702073) failed the inherited case-history API first-page search assertion. The same failure was reproduced on the unchanged base: the route ranked sixth for “case for task.” The isolated route-summary correction from #20370 passed the same three assertions. The search algorithm and assertions were not changed. - No public command changes require a documentation update. The schema and migration record the added index. No hosted DDL or deployment was performed for this PR. - Final head `c50101dabf46ade42b545d2051d724d93f413ccd`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42707623920) passed with 60 successful checks and three configured skips. [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/20372#issuecomment-6752305627) is 5/5 on this head with no open threads. ## Risks - The index adds storage and write cost when a run profile changes. - A concurrent build reads the table and can wait for old transactions. The existing migration path runs it outside a transaction and can repair an interrupted invalid index. - Existing nested-operator lookups can use the index. The companion selector needs this code change to use it. Other JSON path queries are unchanged. - The small local result does not establish end-to-end import speed or PostgreSQL 17 hosted latency. Fresh deployed measurement remains necessary. ## Model Used OpenAI Codex, GPT-6 family. The exact runtime model ID and context window are not exposed in this session. Used reasoning, code editing, shell tools, and local PostgreSQL tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1baac27888 |
feat(server): chat bootstrap reads for instant task opens (#20355)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from the task list to see the latest messages and to reply to an agent > - Before the task chat shows its first frame, it waits for about ten separate reads: the issue, recent comments, comment input facts, recent activity, recent runs, interactions, attachments, live runs and the active run > - On a busy instance each read waits for the database, and the slowest read decides when the chat appears. A cold open takes 1.5–4 s on staging > - This pull request adds one read that returns all of that data for up to ten tasks, in the same JSON shapes as the existing endpoints, plus a single-task form > - The benefit is that the task list can preload its top tasks, and a cold open needs one round trip instead of about ten. The UI change is in the stacked pull request ## Linked Issues or Issue Description **Problem or motivation** A task open waits for about ten reads. With the database under load, these reads queue, and a cold open takes seconds. The list cannot preload a task cheaply, because each task costs about ten requests. **Proposed solution** `GET /api/companies/:companyId/issues/chat-bootstrap?ids=` returns the first-frame data for up to 10 tasks. `GET /api/issues/:ref/chat-bootstrap` returns it for one task. Each field equals the matching single endpoint's JSON exactly. **Alternatives considered** - Prefetch the existing endpoints for each list row. For 30 rows this sends about 210 requests to a database that is already the bottleneck. - A new projection with different shapes. This would duplicate client merge logic, and freshness rules could drift. **Roadmap alignment** This is part of the task-loading performance plan in #20276. It follows #20277, #20284 and #20286. ## What Changed - `packages/shared`: - `ISSUE_CHAT_BOOTSTRAP_MAX_IDS` (10). - `issueChatBootstrapQuerySchema`: UUIDs or identifiers, normalized, deduplicated and strict. - The `IssueChatBootstrap` and `IssueChatBootstrapItem` types. - New `server/src/services/issue-chat-bootstrap.ts`: - Resolves the refs inside the company and applies the `issue:read` decision to each task. - Reads the tasks with a concurrency of 2, and returns them in request order. - `server/src/routes/issues.ts`: - The core issue read and the recent-comments read are now shared helpers, so the bootstrap and the single endpoints use the same code. The core view keeps its Server-Timing header. - Adds the batch route and the single-task route. - Excludes the single-task route from the issue ETag middleware, so a fetch without validators always gets a 200 with `Cache-Control: private, no-store`. - New `server/src/services/issue-run-presence.ts`: - The live-runs and active-run payload builders, moved from `agents.ts` without change. - The existing routes call them, and their behavior is unchanged. - `server/src/routes/openapi.ts`: entries for both routes. ## Verification - `cd server && npx vitest run src/__tests__/issue-chat-bootstrap-routes.test.ts` passes 17 tests. They run on embedded Postgres with privacy enforcement on. They cover: - Each field deep-equals its single endpoint's JSON, for an ordinary task and for a conversation task. - The single-task route equals the batch item, by UUID and by identifier. It is never answered with a 304. - Items keep the request order, with UUIDs and identifiers mixed. - Private, other-company, missing and unknown tasks are omitted, and none of their data appears. The single-task route returns 404 for them. - Malformed, extra or repeated parameters return 400. More than 10 ids also return 400. - A task-bridge key gets 403. - A 10-task request writes nothing: no inserts, updates or deletes, and no activity-log or comment changes. - The issue, agent, activity, live-run, cold-open, summaries, ETag and OpenAPI route suites pass: 56 files, 1,771 tests. - `cd server && pnpm typecheck` passes. - Query cost, local embedded Postgres, one 10-task request: about 49 SQL statements per task, 55–130 ms in total. A test fails if it goes above 60 per task. ## Risks - **Database load:** the stacked UI preloads up to 30 tasks, about 1,500 statements at idle time. This is the same work as opening those tasks, but it is done before the person clicks. The work runs in sequential requests of 10, with 2 tasks at a time. Most of the cost is existing run decoration. Live runs and the active run decorate the same running run, so that work runs twice. Sharing it is a follow-up. - **Stricter access:** live runs and the active run now apply the `issue:read` policy inside this endpoint. The existing routes keep their company-only check. - Hidden tasks are returned, as `GET /issues/:id` returns them. - No migration and no persisted-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing, subagents). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c67909cb0f |
fix: keep tool approvals valid through OAuth refresh and delayed review (#20340)
## Thinking Path > - Paperclip lets people manage agent work and approve app actions. > - An approval must bind the action that the person reviewed. > - OAuth access tokens can expire while that person is away. > - The old target snapshot treated each token refresh as a new authorization. It also imposed a one-hour review deadline. > - This pull request separates token renewal from authorization identity and keeps the human decision durable. > - Current access, the reviewed arguments, meaningful target changes, task closure, and execution receipts still control dispatch. > - People can approve later without repeating a decision after routine connection maintenance. ## Linked Issues or Issue Description **What happened?** A pending app action returned `approved_tool_target_changed` after its OAuth tokens refreshed. The tool and connection settings were unchanged. The same card also expired after one hour. A sign-in problem could discard an accepted decision before provider dispatch. **Expected behavior** The original card remains reviewable after routine OAuth refresh and a server restart. A later approval runs the exact reviewed action once. A connection repair preserves the decision while Paperclip verifies the same authorization. Meaningful changes require a new review. **Steps to reproduce** 1. Connect an OAuth MCP app and configure an action to require approval. 2. Request the action, then refresh its access and refresh tokens before approving. 3. Approve the original card. Before this change, it returns HTTP 409 even when the authorization is unchanged. 4. Wait more than one hour. Before this change, the card expires without a task-specific deadline. **Paperclip version or commit** Reproduced from master at `76254ead2229`. The final branch is rebased onto current master. ## What Changed - Add versioned approval target comparison. New snapshots bind credential identity epochs, consent, permissions, ownership, connection configuration, tool contracts, and credential references. Authenticated OAuth refresh preserves identity. Manual credential replacement does not. - Remove the implicit one-hour deadline from ordinary app action reviews. Keep explicit deadlines and terminal-task expiry. Synchronize the nullable interaction deadline in shared contracts and UI. - Keep accepted, undispatched actions when sign-in is required. Show a saved-approval repair state. Resume on committed connection changes after target and access revalidation, without an idle retry loop. - Preserve the existing execution claim, result receipt, and uncertain-outcome no-replay rules. - Explain meaningful target changes without inviting the person to retry the stale card. - Add a production task-thread Storybook example and document compatibility and recovery behavior. ## Verification - Red: production-refresher invalid-grant handling and formal-approval withdrawal during a sign-in wait also reproduced as failing regressions before their fixes. - Red: before implementation, the local OAuth HTTP fixture failed for both public and confidential clients with `approved_tool_target_changed` after token refresh. - Green: the extended fixture passes for public and confidential clients with personal and shared grants. It covers actual local HTTP token exchanges, multiple refreshes, a rebuilt gateway, an ended originating run, approval after 25 hours, duplicate approval, preserved sharing arguments, sign-in waiting, and event-driven repair. The gateways use the production OAuth refresher; an invalid refresh preserves the accepted decision before provider dispatch. Omitted-scope responses cover equivalent scope-reporting changes. The provider is an isolated local fixture, not a deployed Notebook service. - Negative controls refuse changed consent, scopes, tool definition, destination, and manual credential replacement before provider execution. - Existing gateway service, connected-tool acceptance, secret owner, secret service, unsigned-review queue, and connection-health suites passed. Explicit-deadline fixtures still prove expiry behavior, including deadlines on accepted sign-in waits. A withdrawn formal approval cancels a waiting action before connection recovery. - Approval card, task-chat, error-copy and target-comparison tests passed. Server and UI typechecks passed. Token gates and Storybook catalog checks passed. - Inspected the new saved-approval repair card in local Storybook using the production task thread. It shows a reconnect link and no second approval button or false running state. - `pnpm verify --explain` returned exit 2 and requested broad CI coverage. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507) passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build, typecheck/release registry, general and serialized tests, and all eight browser shards. - Greptile reviewed the same final head at 5/5. Both findings from the first review are fixed and their threads resolved; no open follow-ups remain. ## Risks - No database migration is needed. The new comparison reuses the vault identity epoch. Only authenticated OAuth refresh code can preserve that epoch; HTTP credential-write schemas cannot select that behavior. - Earlier signed requests keep their original exact-version checks and saved deadlines. Already-expired cards are not revived. A historical card whose credentials already changed needs a fresh review. - A new consent, account, permission set, destination, or manually replaced credential must not inherit the old authorization. Reconnection continues automatically only when revalidation proves the same saved authorization. - The interaction enrichment now permits a null deadline. Existing string deadlines remain valid. The web client and shared validator change together. - This PR does not deploy to staging or perform a live Notebook publication. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0e325e8d52 |
fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
79bf733012 |
fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner keeps a lifetime fence until the original ACPX provider stops. > - A recovered Runner must prove that fence is free before it clears saved cleanup state. > - Suspension and other recovery paths already wait for transient listener holders. Recovered shutdown used one immediate probe. > - That difference made parallel Rust tests fail after they dropped their original listeners. > - This pull request uses the existing bounded proof for recovered shutdown and removes the repeated test retry. > - The benefit is one cleanup rule, with the same fail-closed proof and no new work on normal task dispatch. ## Linked Issues or Issue Description Fixes: #20210 Thanks to @genius-0963 for the test-only fix proposed in #20321. That related alternative adds a test-only shutdown retry. This change applies the existing bounded cleanup proof in the shared shutdown path and removes the same retry pattern from native-provider integration tests. It also tests a single shutdown call while the original quorum is still held. ## What Changed - Use `await_provider_lifetime_fence` for a recovered executor with no live session and unconfirmed provider exit. Delete the unused one-shot helper. - Keep the acquired quorum held through the existing durable state write. A failed proof leaves `providerExitUnconfirmed` set. - Give the shared timeout message wording that applies to shutdown and suspension. - Extend the recovery test with a worker-start signal. Hold the original listeners until the single shutdown call is waiting, then release them and assert successful cleanup without a provider session. - Use the same bounded proof in the retired-quorum fixture test. Remove the duplicate retry loop from the native-provider test helper. - Document the bounded recovery cleanup wait. ## Verification - macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only targeted checks ran locally. - `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`: passed. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on the rebased head (5.21 seconds). The initial head also passed five more parallel-test runs (45 tests per run), with no failure. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test native_provider_backend acpx_`: 14 passed on the rebased head (9.43 seconds). These tests use the real Rust executor and controlled sidecars. - Negative control: restore only shutdown's original one-shot proof. The new regression fails at `shutdown must wait while the original lifetime still holds a quorum`. Restore the fix and the affected module passes. - Independent parent verification on current commit `368fc814f7`: the three exact recovery, held-quorum, and retired-quorum tests passed (3 tests, 5.06 seconds). Independent source review confirmed the unchanged three-file scope and found no actionable issue. - The active-turn recovery test still holds a live quorum through the five-second bound. It checks that shutdown refuses cleanup, saved state remains unconfirmed, and the provider never starts. It then releases the quorum and confirms cleanup. - Standard Greptile scored current commit `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads. - Initial GitHub general-server shard 7 failed in unchanged `remote-workspace-nested-verification.test.ts`, at its 15-second wait for a child-process import checkpoint (expected one row, got zero; both Vitest attempts). [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815). That fixture runs Node, PostgreSQL, Git, and storage migration; it does not run the native Runner or the changed Rust cleanup. The cause of the checkpoint timing miss remains unknown. The focused retry passed on the unchanged source ([successful retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)). This proves recovery, not the cause of the original timing miss. GitHub kept the reusable-workflow aggregate failed after that job-only retry and rejected both a final-gate retry and a failed-job retry. A subsequent run-level retry materialized only the gate and aggregate jobs; its aggregate inputs reported prior jobs as `abandoned`, so it failed without a new test failure. A new `full-ci` label event requests a fresh workflow graph on the same commit. The reason for those rerun-state results is not established. - The fresh `full-ci` run [42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208) then lost several runners mid-job. The maintained [runner-loss recovery 42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525) classified all four failed jobs as runner loss: typecheck, E2E shards 2 and 8, and server shard 4. The evidence was a shutdown log or lost-communication annotation. It retried the failed jobs together; canceled sibling jobs were also requeued. This is runner-interruption evidence, separate from the earlier checkpoint assertion. - The maintained recovery attempt again failed the same partial-checkpoint assertion in unchanged `remote-workspace-nested-verification.test.ts`, twice ([job 129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)). Neither checkpoint-failure job logged runner shutdown or lost communication; their resource summaries reported no significant constraint. The child phase at the timeout remains unknown. No further blind retry or TypeScript change is included here. Related startup-handshake work exists in #20339; its controlled-delay evidence establishes a deadline-boundary defect but does not establish the phase of these original CI failures. - Rebased onto current master after upstream #20339 merged. That separate change fixes a demonstrated child-startup/checkpoint-deadline boundary. It does not prove which child phase caused the earlier zero-checkpoint assertions. This PR still changes only its original Rust and documentation files. Fresh GitHub CI on the new base will establish recovery, not retrospective cause. - Full local tests and a local release build were not run. Current-head [PR CI 42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411), [Runner Windows 42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803), and [Docker Runner 42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601) all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check list has 59 successes and 5 intentional skips. GitHub supplied broad typecheck, build, and optimized Rust coverage. This is new-base recovery; the cause of the original checkpoint assertion failures remains unproved. ## Risks - A genuinely unconfirmed original provider can now hold recovered shutdown for up to the existing five-second proof deadline. The proof returns immediately when two candidate ports are free. - The changed branch requires both no in-memory provider session and unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch keep their existing paths. Outer controller close deadlines remain authoritative. - State saved by the previous release carries forward unchanged. No schema, launch identity, profile digest, protocol, or retry-budget change. - No live provider session or hosted deployment was tested. The tests cover cleanup proof, refusal, durable state, and controlled provider execution. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
4a349ea447 |
fix(chat): tell agents that instruction saves keep no history (#20328)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents can read and edit other agents' instruction files with dedicated tools > - #14420 changed instruction saves: Paperclip now keeps only the current file and records no revision history > - The agent-facing descriptions still describe a "canonical instruction revision" history > - In a live eval, a chat agent asked to undo an instruction change found an empty history, read it as a test artifact, guessed the earlier text from the current file's wording, wrote it, and reported a restore > - This pull request tells agents the real contract in chat guidance and in the tool and API descriptions > - The benefit is that an undo with no saved version leaves the instructions alone and asks the person, instead of inventing an earlier version ## Linked Issues or Issue Description Refs #14420 Refs #19515 (open: would list save records from the activity log. Those records hold no content, so they still cannot restore earlier text) **What happened** A person asked a chat agent: "Undo Morgan's last instruction change." Morgan's history was empty, because saves do not record revisions. The agent's reasoning said the history "is only populated with canonical revisions" and that there was "no revision tracking in this e2e harness". The agent then removed the last line of AGENTS.md and replied "Undone. Morgan's AGENTS.md is back to …". No saved source had the earlier text, so the agent guessed it. **Expected behavior** When there is no saved earlier version, the agent leaves the instructions unchanged. It says plainly that no earlier version was saved, and it asks the person for the text they want or which part to remove. **Steps to reproduce** 1. Create an agent with an AGENTS.md file. Edit the file once through `PUT /api/agents/{id}/instructions-bundle/file`. 2. `GET /api/agents/{id}/instructions-bundle/history?path=AGENTS.md` returns `{"revisions":[]}`. 3. Ask a chat agent to undo the agent's last instruction change. The agent rewrites the file from a guess. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says three things: - Saves keep only the current instruction files. - An empty history means no earlier text is retained. - An undo with no listed revision must leave the file unchanged, say that no earlier version was saved, and ask for the wanted text. - The `get_agent_instruction_history` contract now says that it lists only pre-upgrade revisions, and that an empty list means Paperclip retains no earlier version. - The `restore_agent_instructions` contract now says that only listed pre-upgrade revisions can be restored. - The OpenAPI summary for `GET /api/agents/{id}/instructions-bundle/history` says the same. - The semantic tool contracts, the protocol manifest and the seeded eval catalog hash are regenerated. - Regression tests check the new guidance and both live tool contracts. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts server/src/__tests__/board-access-denial.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/runner-api.test.ts` passes. - In `packages/paperclip-runner`, `pnpm exec vitest run src/semantic-tools/paperclip-semantic-tools.test.ts src/catalog src/evals/native-execution.test.ts` passes. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-protocol-manifest.mjs --check` and `generate-replay-goldens.mjs --check` pass. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - The live result is the chat eval `agents.restore-instructions`, run with this change cherry-picked: run 42666621831 passed. The agent replied that Morgan's history is empty and no earlier version exists. It left AGENTS.md unchanged and asked for the exact text or the line to remove. Run 42660236086, without this change, failed: the agent guessed and rewrote the file. ## Risks - Low risk. Only agent-facing text changes, and no behavior changes. - The runner semantic catalog hash changes. This is the same kind of change as #19983. - If #19515 lands, history lists save records without content. The guidance wording must then say that listed saves without content cannot be restored. Whichever PR merges second must reconcile the two texts. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
76254ead22 |
fix(runner): allow Codex interruption to settle governed waits (#20330)
## Thinking Path > - Paperclip manages AI agents and their work. > - A native run must stop provider work before it waits for a human response. > - The Codex transport allows 30 seconds for a control command. > - The governed-wait loop used its 15-second default for Codex. > - A valid interruption terminal at 18 seconds therefore failed the run. > - This change gives Codex the existing 30-second terminal drain allowance. > - The run still fails if it cannot prove that the provider turn ended. ## Linked Issues or Issue Description **What happened?** A Codex run can save a human wait, accept an interruption, and fail before the provider sends its terminal. Its saved wait remains intact, but normal finalization does not occur. **Expected behavior:** Allow terminal delivery through the transport's bounded command window. Keep missing-terminal cases closed. **Steps to reproduce:** Run the new protocol regression against the old backend. The real Runner binary uses a controlled Codex process that delays its interruption terminal for 18 seconds. The run fails at the 15-second drain deadline. The same process passes with this change. **Paperclip version or commit:** Reproduced against `1b7fe33432`. **Deployment mode:** Local test of the native Runner path. The incident was observed in a hosted sandbox. **Installation method:** Source checkout. **Operating system:** macOS arm64 for local verification. Hosted incident runtime was Linux. **Additional context:** No matching repair was found. This addresses a reproducible deadline mismatch. The hosted log ends terminal observation at 15 seconds, so it cannot prove whether that provider would have settled within 30 seconds. The approved local live-provider check passed. This PR does not claim recovery of the hosted task. ## What Changed - Give Codex a 30-second governed-wait terminal drain through the existing backend option. - Keep other providers and the caller's shorter execution bound unchanged. - Add real Runner protocol cases for prompt, delayed, and absent terminal delivery. Check retained wait state and one provider turn. - Document the bounded wait and terminal-proof requirement. ## Verification - Red: the 18-second controlled-provider case failed with `governed_wait_provider_terminal_unproven` on the old backend. The prompt case passed. - Green: 282 tests passed across the native runtime, harness backend, and new Runner protocol suite, including a repeat on the rebased PR revision. `pnpm verify --explain` returned exit 2 because the execution-contract document requires broad CI; the [required CI run](https://github.com/paperclipai/paperclip/actions/runs/42668235142) passed on the final head. All 58 applicable checks passed; the three skipped checks were the two unrelated Storybook jobs and the selected-test planner replaced by broad coverage. - The missing-terminal case still fails without `completeRun` or a successful run terminal. The saved governed wait remains available. Only one provider turn and one tool call occur. - Existing runtime tests cover shorter deadlines, foreign or unproven terminals, complete usage without a terminal, and checkpoint recovery without another model turn. - Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64, disposable local workspace. One `request_human_input` tool call returned a pending interaction fixture. The real provider emitted `turn.interrupted`; the controller accepted the yielded wait and its run terminal. The check passed in 8.14 seconds. The tool callback and persistence port were controlled; no hosted server permission or Notebook publication claim is made. - Initial local probe setup failed before model execution: first the explicit launch lacked `app-server`, then it omitted the Runner permission configuration. The successful check used the Runner's normal launch arguments and isolated home with the approved existing account. - Hosted sandbox acceptance and recovery remain unverified. The 18-second delayed shutdown is a controlled protocol regression; the live model check proves normal shutdown, not that specific latency. ## Risks - A Codex provider that never stops can occupy the settlement window for up to 15 additional seconds. It still fails closed and is quarantined. - No persisted schema, provider identity, permission, or recovery rule changes. Previous-release checkpoints use the same format and existing recovery checks. - This change cannot fix a provider that never emits a terminal. It also does not grant missing setup permissions or decide pending approvals. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool calls, code edits, and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
a3e7e63330 |
Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. **The current fix has a causal red/green check.** On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: **5 passed, 0 failed;170 filtered rows excluded**. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed **25/25**, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, **15 selected accepted-result/restart controls passed**. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete **17-case maintenance-server file**. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed **29/29**. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
8fe524f796 |
fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a new attempt can start on a task, Paperclip must prove the previous run's agent stopped, so two agents never work the same task at once > - A local native run can be interrupted after it asked to start the runner but before it recorded the runner's process. The runner may have saved state and started a provider, and provider processes can outlive it, so the existing check (#19694) rightly refuses to assume the run stopped > - Today that case has no way out: every Retry is accepted and then waits forever as `process_identity_missing` (storage chaos ST1, t02/t03) > - This pull request makes the wait explicit and actionable. The task explains why it waits, and a board operator who has checked can confirm the run stopped; the parked Retry then starts at the click > - A second shape from the Daytona storage baseline (t06) parked a Retry with no reason at all: a follow-up was interrupted by a graceful shutdown while it was still restoring its workspace. No provider ran, but the restore hold kept the same agent held, so the Retry never started > - The benefit is that no Retry stays accepted-and-waiting with nothing anyone can do ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks t02 and t03. **What happened (CHA-3, run `3c3f6241`):** - The run requested a runner launch (`native.process_start_requested`), and a K2 restart hit before its process identity was recorded. - After the restart, the resumed attempt failed on the storage outage. The task was blocked. - The board's Retry (202) was parked as `executionWait.reason = process_identity_missing`: "The previous run has no verified stop record." - `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this run's runner saved control-plane state. Once that exists, a provider may have started (#19694), and a provider is not guaranteed to carry `--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on purpose). The shared owner prover needs a PID, which the run never recorded. - Nothing could ever clear the wait. **Daytona storage baseline `w-20261109T154810-s9911`, task t06:** - The run succeeded. The follow-up was cancelled (`continuation_task_ownership_changed`), and the next run was interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s into `preparing`. It was restoring the saved remote workspace and had launched no provider. - Restore recovery saved the files (`workspaceRecovery.status: recovered`, system recovery `complete`). But the `legacy_execution_requires_reconciliation` restore hold releases only for a different owner, or for a native Stop fenced before any provider turn, so it kept the same agent held. - The UI Retry returned 202 and parked with `executionWait.reason: execution_recovery`, the generic wait, so the task showed a Retry that never started. Automatic proof needs the runner to record its identity before saving state and to group provider processes under it. That is follow-up runner work; this PR removes the endless wait now. ## What Changed - `server/src/services/native-local-process-stop.ts`: new `confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished native run of the task with no recorded process and whose latest launch evidence is an unconfirmed launch on this host in a local environment. It still refuses while a runner with `--run-id <run>` is running. It records `native.local_process_stopped` with `basis: "operator_attested"` and the attesting user. That is the receipt the existing continuation gate already accepts. - `server/src/routes/issues.ts`: `POST /issues/:id/recovery-actions/confirm-run-stopped` (`runId`, `stopEvidence` of 20 to 12,000 characters). It is board-only and requires issue read plus `runtime:confirm_owner_stop`, like the other stop attestations. It logs `issue.run_stop_confirmed` with the evidence, promotes deferred intents for the issue, and resumes each parked task Retry, so work starts at the click. Refusals return 409 with a code. - `packages/shared`: `confirmRunStoppedSchema`, and the presentation user action `confirm_run_stopped`. - `server/src/services/task-interruption-presentation.ts`: a Retry parked as `process_identity_missing` for a native run that recorded no process now presents as waiting for the user (`canRetry: false`, `userAction: "confirm_run_stopped"`), with a message explaining why and who can act. A run with a recorded process keeps the existing verification wait. - UI: - `ExecutionBlockerNotice` shows the explanation and, for users who can confirm a stop, "Confirm it stopped" with a short evidence form. - It reuses `PreviousServerStopConfirmation`, now with optional `action`, explicit `runId` and `formId`, and a custom button label. - The Inbox names the action. - `server/src/services/execution-blocker.ts`: `interruptedPreparationBeforeProviderWorkCondition`. The restore hold also releases for the same agent when the stopped run was a system-interrupted preparation (legacy runtime never resolved, still `preparing`) with no process identity, no runner or provider events, no finalization or result, and every lease released with a confirmed stop. It mirrors the existing interrupted-preparation proof and still requires the completed workspace-recovery receipt. Auto-replay is unchanged; this only lets the user's Retry or message start. - Review fixes: - `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and lease checks, and the presentation offers `confirm_run_stopped` only when it passes. - The attested stop wakes cost accounting in the same transaction. - The confirmation form is keyed by task and run. - The route is registered in OpenAPI. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`: - The parked Retry presents `confirm_run_stopped` with the explanation. - Confirming records the operator-attested stop and the activity entry, satisfies the stop check, and resumes the parked Retry: its receipt is coalesced into a newly started run. - It refuses with 409 while a runner for the run is running. - Agents and viewers get 403. - `ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`: the notice offers "Confirm it stopped" (and no dead Retry) and posts the run and evidence. A viewer sees the explanation without the action. - `server/src/__tests__/interrupted-preparation-restore-retry.test.ts` (t06 shape): - The restore hold no longer blocks. - A Retry is accepted and resumed into a started run, and the audit hold keeps `replay: blocked`. - The hold stays when a provider turn started, when a runner event was recorded, when an operator stopped the run, or when the sandbox has no stop receipt. - `issue-confirm-run-stopped-routes.test.ts`: - A case per refusal code checks the presentation keeps the generic wait and the endpoint returns 409. - An `alreadyStopped` case. - An accounting-wake assertion. - The UI test switches targets with stale evidence. Regression and static checks: - `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control tests: 74 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A board operator can now attest that an unidentified run stopped. The attestation is permission-gated, requires written evidence, is activity-logged, and is refused while a runner for the run is visibly running. A wrong attestation could start a second agent beside an orphaned provider, the same trust model as the existing previous-server attestations. - The confirmation component now accepts a run without a recovery action. Existing callers pass the action as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ff2d463eed |
feat(cli): manage agent grants and keep remote login on the public origin (#20323)
## Thinking Path
> - Paperclip lets people control their AI agents and permissions.
> - Remote operators need a browser-approved CLI login.
> - Proxies can send that login's approval link to an internal host.
> - Human member commands cannot grant an agent an explicit permission
such as commentary:view.
> - This change keeps approval on the canonical public origin and adds
Board-only single-grant commands.
> - The API checks company access and permission-management authority
and records each change.
> - An operator can sign in, grant one agent permission, and preserve
the agent's other grants.
## Linked Issues or Issue Description
**What happened?**
A proxied CLI login returned an approval URL on the upstream host. The
signed-in browser was on the public host. The CLI also had no command to
add an arbitrary explicit agent grant: member commands manage human
members, while agent permissions:update exposes fixed configuration
fields.
**Expected behavior**
Approval uses the configured canonical public origin. A Board operator
with users:manage_permissions can grant or revoke one explicit agent
permission without replacing other grants or activating an inactive
membership.
**Steps to reproduce**
Configure a public auth origin. Create a CLI challenge through a proxy
that replaces forwarded-host headers. Inspect the approval URL. Then try
to grant commentary:view to an agent through the CLI.
## What Changed
- Use the existing public-origin resolver for CLI approval URLs.
- Add agent permission:grant and permission:revoke commands.
- Add PATCH /api/agents/{id}/permission-grants with strict shared
validation and Board-only OpenAPI metadata.
- Serialize each mutation on the agent row. Preserve unrelated grants
and membership state. Commit the grant and its activity record together
and publish activity after commit.
- Document remote login and single-grant management.
## Verification
- The new approval-link regression failed before the fix and passes
after it.
- All 25 CLI auth route tests pass.
- All 72 affected CLI, login and OpenAPI tests pass.
- All 5 PostgreSQL grant tests pass, including actual commentary reads
(403 before grant, 200 after grant, 403 after revoke), preservation,
audit attribution, concurrent additions, invalid inputs, inactive or
missing membership, agent self-grants, cross-company denial, and
rollback on audit failure.
- Shared and CLI typechecks and server declaration-layer checks pass.
- pnpm verify requests broad CI coverage because doc/CLI.md is a global
verification input. Greptile reviewed final head
|
||
|
DottaandClaude Opus 5.5
|
97188f276c |
perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.
## Linked Issues or Issue Description
Refs #20190, #20285
**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).
**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.
**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.
## What Changed
- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
- New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
- The other `DurableRunnerConfig` literals set `turn_scoped: false`.
## Verification
- `cargo test -p paperclip-runner-core`:
- `--lib`: 602 passed.
- `--test acpx_park_preparation`: 22 passed.
- `acpx_park_readiness`: 8 passed.
- `acpx_warm_attachment`: 4 passed.
- `acpx_provider_suspend`: 5 passed.
- `acpx_provider_session`: 12 passed.
- `acpx_provider_turns`: 29 passed.
- `native_provider_backend`: 47 passed.
- `durable_recovery`: 3 passed.
- `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(
|
||
|
DottaandClaude Opus 5.5
|
fd2e7482c0 |
fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop
## Linked Issues or Issue Description
Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).
**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.
**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.
**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
e015c2ef21 |
fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path - #20145 prepares the next Claude provider generation at park, and #20190 restarts the ACPX provider when a turn settles. They move the sidecar respawn off the follow-up's critical path. - When that turn-end restart fails, the runner "fails closed": `restart_session_after_turn` saves the session `closed` and drops it. Within the run, the next turn fails exactly like a failed restart at turn start. That's intended. - Across runs it breaks the conversation: - the next run's `run.attach` refuses a `closed` session ("run.attach requires the same settled ACPX provider profile and session"); - it reconnects for about 10 s; - the controller then records a continuity break and starts a fresh Claude conversation (`run.startup.reuse provider_session: fresh/created`). - A local A/B (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`, on a host with load average about 17–27) confirmed it: | | |
||
|
Nicky LeachandPaperclip
|
09412e652a |
fix(runner): allow warm release to drain after provider restart (#20290)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm Runner sessions park after a turn so follow-up tasks can claim the same process. > - Claude restarts its ACPX provider after the controller receives the terminal result. > - That restart can delay the drain receipt beyond the release path's one-second limit. > - A failed drain prevents parking, so the idle timer later closes the retained Runner. > - This pull request uses the existing warm admission budget for release drain. > - The benefit is retained Runner reuse without moving provider restart onto task dispatch. ## Linked Issues or Issue Description Fixes: #20225. Refs #20190 and #20145. Related performance follow-up: #20224. ## What Changed - Align the release drain timeout with the existing 30-second warm admission budget for every provider. Close paths keep their explicit remaining deadlines. - Keep the existing immediate completion when the drain receipt proves settlement. Add no sleep, retry, provider branch or dispatch work. - Add a real Runner regression with a fake ACPX sidecar whose replacement start takes 1.5 seconds. One release call must park the Runner, then a follow-up must claim the same Runner process. - Document the release drain budget. ## Verification - Targeted command in `packages/paperclip-runner`: `vitest run src/live/runnerd-codex-transport-release-resume.test.ts src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed** across 3 files, 157.78 seconds. - Independent parent verification reran the slow-restart and ordinary cached cases: **2 passed**. Source review confirmed that release is the only caller using the changed default; close callers supply explicit deadlines. - Negative control: with the original one-second timeout, the new test fails because the single release call returns `false`. The fixed code passes the new case. - Built only the current-source Runner and two fixture binaries required by these tests. The first sandboxed invocation could not bind its loopback listener (`EPERM`); the test ran with loopback access afterward. - Full GitHub PR workflow [42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993) passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard Greptile scored this same head 5/5 with no review threads. Full repository checks ran in GitHub CI, per the requested local test scope. No live Claude session or hosted deployment was tested. ## Risks - An unresponsive Runner can now hold release drain for up to the existing 30-second warm admission limit. A healthy Runner returns as soon as its receipt arrives. The provider restart stays at turn end; task dispatch gets no fixed delay. - Saved state, protocol and provider identity formats do not change. Previous-release state carries forward under the existing validation. - The controlled fake-sidecar test proves the restart timing race and process reuse. It does not qualify live provider latency or a hosted rollout. ## Model Used - OpenAI GPT-6 through Codex, with repository tools, code editing and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1b09920a64 |
fix(issues): refuse agent status changes that detach another run's live execution (#20271)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that an agent run executes holds a checkout lock and an execution lock (`checkoutRunId`, `executionRunId`) > - A status change away from `in_progress` clears these locks. It does not stop the run that holds them > - Agents cannot stop another agent's run: run cancel and the task-tree pause are board-only > - But an agent that can edit a task (for example a lead with the manage-active-checkouts grant over its report) can change its status. The run then keeps running and is no longer bound to its task > - In chat evals, a lead "paused" a launch tree. It moved running subtasks to backlog. The runs kept running, detached > - This pull request refuses an agent's status change when a different queued or running run holds the task's lock. The denial is final, links the task, and names the person's controls > - The benefit is that agents cannot interrupt another agent's run through a status edit, and runs are not left orphaned ## Linked Issues or Issue Description Refs #20261 (chat guidance sends a task-tree pause to the person's Pause subtree control). Refs #19394 (resume restarts the work a subtree pause interrupted). **What happened** An agent changed the status of a task while another agent's run executed it. The request succeeded. The server cleared `checkoutRunId` and `executionRunId`. The run continued, but it was no longer bound to the task. In the chat evals `steering.stop-versus-pause` and `task-coordination.pause-tree`, a lead moved running subtasks to backlog to "pause" a launch. Their runs kept running, detached. **Expected behavior** An agent cannot get the effect of interrupting another agent's run. When a different live run holds the task's lock, the agent's status change is refused before any write. The refusal says it is final. It sends the person to the task and names the controls that really stop or hold the work: the run's Stop or Interrupt, or More task actions, then Pause subtree on the parent task. The lock-holding run, board users, and paths that stop the run themselves continue to work. **Steps to reproduce** 1. Create agent A and agent B, with B reporting to A (A gets `tasks:manage_active_checkouts` through the manager chain). 2. Give B an `in_progress` task. Set `checkoutRunId` and `executionRunId` to a `running` heartbeat run of B. 3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with `{ "status": "backlog" }`. 4. Before this change: `200`, the status is `backlog`, both lock fields are `null`, and B's run is still `running`. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it refuses a status change when `executionRunId` or `checkoutRunId` names a different run in the same company that is `queued` or `running`. It runs in `PATCH /issues/:id` after the final status and assignee are known (execution-policy transitions included) and before any run stop or write. It also runs in recovery resolution when the source status changes. - The guard applies to any status other than `in_progress`, also to the current status, because the service clears the locks for those too. - The write is conditional on the lock the guard checked. The route re-reads the lock and passes it to `svc.update` as `expectedExecutionLock`. The service compares it under the row lock and returns `409 issue_execution_lock_changed` if a checkout landed in between. - These cases stay allowed: the lock-holding run itself, board users, requests with no status, a change to `in_progress`, `cancelled` (the route cancels the run), reassignment (the route stops the run first), `done` through PATCH while a Runner goal is active (terminalization pauses the goal and stops the run; recovery resolution does not get this exception), and a lock holder that already finished. - `packages/shared/src/issue-write-denial.ts`: new denial code `issue_status_change_live_run` (409). The copy says the refusal is final, links the task as `[identifier](/prefix/issues/identifier)`, and names Stop or Interrupt and More task actions, then Pause subtree. A new optional `issuePath` context field carries the link. The UI notice renders it through the existing generic path. - `doc/SPEC-implementation.md`: documents the rule in the checkout contract section. - Paths checked with no change needed: there is no bulk issue update route; tree holds and run cancel are board-only; `POST /issues/:id/release` already lets only the assignee's checkout run release; API-catalog actions call the same PATCH route; MCP status changes already refuse with `MCP_ACTIVE_EXECUTION`. ## Verification - Red on master, green with this change: - `server/src/__tests__/issue-stale-execution-lock-routes.test.ts` (embedded Postgres, real issue service): a manager moves its report's running task to backlog. On master: `200`, status `backlog`, `checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409 issue_status_change_live_run`, row unchanged, run still `running`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued lock holder, and a repeated current status (red on master). Allowed cases: field edits, the lock-holding run, board, `cancelled` (run is cancelled), and finished lock holders; an allowed change carries `expectedExecutionLock`. - Same embedded-Postgres file: `svc.update` with an `expectedExecutionLock` that differs from the row returns `409 issue_execution_lock_changed` and leaves the row unchanged. - Also passed locally: `issue-closed-workspace-routes`, `issue-comment-reopen-routes`, `issue-recovery-actions`, `issue-scheduled-retry-routes`, `low-trust-red-team-routes`, `issue-update-comment-wakeup-routes`, `external-object-routes`, `packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`. - `tsc --noEmit` shows no errors in the changed files. Full typecheck, build and test are left to CI. ## Risks - Behavior change for agents: a lead or CEO that used status edits to take over a report's running task now gets a 409. The supported paths (comment, cancel, reassign, or the person's Stop or Pause subtree) still work. - A queued run that holds the lock also blocks the change. This is deliberate: the queued run would otherwise start later with no task bound to it. - Agents can still cancel another agent's run by cancelling its task. That is existing, documented behavior ("task cancellation cancels the intent"), and this PR does not change it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3374ca54f5 |
fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8eb11fa8aa |
fix(runner): fail closed with a recorded cause when runner event commits stall (#20235)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Runner streams numbered events to the control plane, which commits each one and acknowledges the committed prefix > - In restart chaos wave `w-20261109T093417-s9807`, t01 (claude R1, local), the control plane never committed Runner seq 34. Every failure path in `#event` closed the connection without recording why. The Runner reconnected 2,848 times over 14 minutes and nothing detected it > - This pull request makes that loop a detected, named failure that carries its cause > - The benefit is that a wedged event stream becomes a visible run failure with a reason, instead of a silent "running" task ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T093417-s9807` (master `df4562ab38`), task t01, run `d08dc899`. **What happened** (Phase 2's re-check of the kept instance): - The provider kept working. Claude's transcript runs to 09:47:03 and ends in `paperclip_finish`. - The Runner, its ACPX provider state, and the controller journal agree on acked seq 33. The controller never commits Runner seq 34 (Write progressed, 09:36:25.892). - Phase 2 replayed the kept outbox offline. Runner seq 34 fails `validatePrpEvent`: its `/payload/target` contains a newline from a malformed Claude Write call. `#event` closes without a reason, and the Runner redelivers the same bytes forever. (Phase 2's separate PR stops the producers from emitting such paths.) - Every `#event` failure path closed the connection without recording a reason, so the cause was invisible. An outbox that already holds such an event can only be cleared by failing closed. - The `<instance>:<run>:N` ids in `heartbeat_run_events` come from the TS driver's own projection numbering, so there was no Runner sequence rewind. ## What Changed - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: - `#eventCommitFailed` replaces the bare `connection.close()` on each `#event` commit-failure path. It records a bounded cause: `event_sequence_mismatch` for an out-of-order event, `event_schema_invalid:<issue paths>` for an event that fails `validatePrpEvent`, the error's code or name (never its message) for a commit callback error, `journal_sequence_not_contiguous`, `journal_commit_refused`, or `checkpoint_flush_failed:<cause>`. - If failures continue for the same run at the same acknowledged cursor for at least 20 attempts and 45 s, it fails closed with `NativeSessionProtocolIntegrityError("runner_event_commit_stalled", "<cause> at source seq N")`, where N is the failing event's own sequence. Only a moved acknowledged cursor counts as progress (a replayed, already acknowledged duplicate does not), and a different run starts a fresh counter. A single transient failure still just reconnects, as before. - A checkpoint flush that fails before any outbound frame also counts, as `checkpoint_flush_failed:<cause>`. That includes the authentication challenge and the welcome, so a reconnect that cannot even authenticate still reaches the bound. - A journal write that throws (`#store.commit`, `#store.commitEvent`, or the welcome save) leaves the store unwritable until reload, so it fails closed at once with `journal_write_failed:<code>`. - `packages/paperclip-runner/src/contracts/native-session-backend.ts`: new reason `runner_event_commit_stalled`, with an optional `detail` in the message. ## Verification Red/green: - "bounds commit-stall reconnects for an out-of-order event": a schema-valid event with the wrong `sourceSeq` fails closed with `event_sequence_mismatch at source seq 5`. Before this it closed silently forever. "… for 20 failures inside 45 s" pins the time bound: 20 failures over 19 s do not fail closed. - "fails closed when checkpoint storage keeps failing before a reconnect authenticates": with checkpoint persistence refusing (`ECONNREFUSED`), 20 reconnect attempts fail before authentication, then the control plane fails closed with `checkpoint_flush_failed:ECONNREFUSED at source seq 1`. Without this, those attempts close silently. - "fails closed at once when the journal cannot record a runner event": a journal write throwing `ENOSPC` fails closed with `journal_write_failed:ENOSPC at source seq 1`, and the private error text is not carried. On master the connection only closes. - `durable-prp-control-plane.test.ts` "fails closed with the schema issue when the runner keeps redelivering an invalid event (restart chaos t01)": an event that fails `validatePrpEvent` is redelivered 20 times over 57 s of mocked clock. On master the control plane only closes; with this PR it fails closed with `runner_event_commit_stalled` and detail `event_schema_invalid:/… at source seq 1`. - `durable-prp-control-plane.test.ts` "fails closed when runner events keep failing to commit without progress": 20 reconnects over 57 s of mocked clock, each commit failing with `ECONNRESET`. On master nothing fails closed. With this PR the 20th failure reports `runner_event_commit_stalled` with detail `ECONNRESET at source seq 1`, and the private error message is not carried. File: 201 passed, including the existing "keeps a non-typed commit failure retryable" cases. Static checks: - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. The server is unchanged. ## Risks - Storage that keeps failing event commits for at least 45 s and 20 attempts now fails the run (operator recovery) instead of looping silently. Shorter outages are unchanged. - The recorded cause is a code or error name only, never error text, so no tool content or credentials reach the run error. - No schema change and no change to persisted state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3f2a940d5b |
fix(runner): answer held Paperclip tool calls after a local restart before the next turn (#20227)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart interrupts a turn, the Runner keeps each Paperclip tool call the model made as "held" until Paperclip answers it, and it refuses to start the next turn while any call is held > - Paperclip answers held calls after a reattach through its tool recovery path. That path was wired only for remote runners > - In restart chaos wave `w-20261109T084023-s9403`, t101 (local codex, K2 then K1) held a `read_document` call that nothing answered, so the continuation's `turn.start` failed and the task needed a Retry > - This pull request wires held-call recovery for local runners too, and makes `turn.start` answer or wait for held calls first > - The benefit is that a turn lost to a restart no longer strands the run behind one unanswered call ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T084023-s9403` (master `fc5c79a327`), task t101 (codex C1, K2, local). **What happened:** - Run `4b5810d5` called `read_document` (`exec-ac8fa9b4`) around the K2 drain at 09:08:18. A K1 then killed the next server start. - At 09:08:36 the third controller reattached. The Runner restored the conversation and reported the old turn lost (`provider_turn_lost_on_restore`, recoverable). The held call stayed pending, by design: a held call accepts a late result. - The controller had the call: the quarantine detail lists `exec-ac8fa9b4` under `semanticTools.pending`. But `recoverPendingSemanticInputs()` returned at once: for a local run `controllerPersistence` is unset, so `recoverSemanticToolInput` was `undefined`. - At 09:08:46 the continuation's `turn.start` failed with "cannot prepare the next provider turn while tool calls are active (pending: 1 [read_document:exec-ac8fa9b4…])". The run ended `native_session_interrupted`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: the runnerd transport now always gets `recoverSemanticToolInput`, not only when `controllerPersistence` (remote) is set. It uses the same epoch check and `SessionAuthorityEpoch.recover`. Recovery re-authorizes and re-runs reads, and replays receipts for mutations; it never repeats an uncertain mutation. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts` (`#startTurnWork`): when the control plane has unsettled semantic inputs, it runs `recoverPendingSemanticInputs()` before queueing `turn.start`, and waits for that pass, bounded by the turn-start deadline. Recovered results are queued as commands ahead of `turn.start`, and the Runner applies commands in order. If recovery has not finished by the deadline, the turn start fails with `native_turn_start_held_tool_calls_unresolved` instead of sending `turn.start` behind it. When everything is settled there is no wait, so a normal turn start is unchanged. Live calls of a turn being replaced are not waited on; the replaced start still rejects them. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: `recoverPendingSemanticInputs()` passes are serialized. A call made while the startup pass is still answering a held call waits for that pass, then runs its own, so the turn-start gate cannot finish early by skipping a call that is already being recovered. ## Verification Red/green: - `native-session-executor.test.ts` "answers a local run's held Paperclip tool calls after a restart (restart chaos t101)". On master `recoverSemanticToolInput` is `undefined` for a local run; with this PR it is wired. - `runnerd-codex-transport.test.ts` "answers a held call before starting the next turn (restart chaos t101)", against real runnerd and fake Codex, with a slow startup recovery pass already in flight when `turn/start` arrives. On master `turn/start` never ran recovery, and without the serialization `turn.start` is queued before the held call's result. With this PR the held call is recovered once and its result is queued before `turn.start`. Regression and static checks: - `runnerd-codex-transport.test.ts`, `durable-prp-control-plane.test.ts`, `runnerd-codex-transport-adoption.test.ts`: 510 passed. - `native-session-executor.test.ts` and `native-session-resume.test.ts`: 1368 passed, 1 failed. The failing case, "archives a damaged prior epoch … with real runnerd", fails identically on master on this Mac (`ENOENT … authority-epochs`), so CI is its reference. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Local runs now recover held calls after a reattach, as remote runs already did. Reads re-run under current authority. A mutation without a saved receipt still returns no result and stays held, as before. - `turn.start` may now wait up to its existing deadline while a recovery pass runs, then fails clearly if it has not finished. It never waits when nothing is unsettled, so normal turns get no added latency before `turn.start`. - No schema change and no change to persisted run state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
465da4227a |
fix(runner): drop a tool target path that breaks a line so the controller can acknowledge its event (#20238)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner turns provider tool calls into durable PRP
events. The controller acknowledges each event only after it validates
and commits it.
> - The controller's event schema forbids line terminators in a safe
path such as `tool.execution` `target`.
> - The ACPX sidecar and the Rust runner accepted a location path that
contained a newline. Claude sometimes emits a malformed Write call that
puts the file content into the path.
> - The controller refused that event without an acknowledgement. The
runner redelivered the same bytes, and the run stopped making progress
for 14 minutes (2,848 reconnects).
> - This pull request makes both producers drop such a path, so the
event carries a null target and the controller accepts it.
> - The benefit is that a malformed provider path can no longer wedge a
run's event delivery.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9807 (claude, R1, local), the run stopped
receiving events after runner source seq 33. The runner reconnected
2,848 times in 14 minutes. Offline replay of the saved runner outbox
through the controller's `validatePrpEvent` showed that seq 34 fails
`/payload/target` (the safePath pattern), because the target is
`packages/shared/src/chaos/slug-2c5c3c8f.ts\n<parameter
name="content">export function chaosSlug…`. Seq 35, 39 and 40 (the same
tool call) fail the same way.
**Expected behavior**
The runner never emits an event that the controller's schema refuses. A
malformed tool path becomes a null target, and the tool event still
arrives.
**Steps to reproduce**
1. Have an ACPX provider send a tool call whose location path contains a
newline (for example, a Write call with the content in the path).
2. See that the runner emits `tool.execution.progressed` with that
target.
3. See that the controller closes the connection without an
acknowledgement on every redelivery.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
d2fe85e57c |
fix(secrets): delete a secret that a binding proposal referenced (#20193)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose a secret binding. A person approves or rejects it on a review card. > - Each binding proposal row refers to one source: a secret or a pending secret proposal. A check constraint enforces this. > - The secret reference used `ON DELETE SET NULL`. When a person deletes the secret, Postgres sets the reference to null. The check constraint then rejects the row. > - So a person cannot delete any secret that a binding proposal ever referenced. The API returns 500. > - This pull request deletes the proposal rows together with the secret, as the other source reference already does. Before the delete, it rejects pending proposals so their cards close with a reason. > - The benefit is that secret deletion works again and no review card points at a removed proposal. ## Linked Issues or Issue Description Refs #15644 (related: block secret deletion while bindings still exist). **What happened** `DELETE /api/secrets/:id` returned 500 for a secret that a binding proposal referenced. The server log showed `new row for relation "company_secret_proposals" violates check constraint "company_secret_proposals_shape_check"`. This happened for pending, rejected and approved proposals. It also happened after the person removed the agent's binding. **Expected behavior** The person can delete the secret. Proposals that referenced it go with it. A pending review card closes as rejected and tells why. **Steps to reproduce** 1. An agent proposes a binding for an existing company secret (`POST /api/agents/me/secret-proposals` with `secretId`). 2. Approve or reject the proposal, or leave it pending. 3. Delete the secret with `DELETE /api/secrets/:id`. 4. The response is 500 and the secret stays. ## What Changed - `packages/db`: `company_secret_proposals.secret_id` now uses `ON DELETE CASCADE`, the same as `secret_proposal_id`. Migration `0452_secret_proposal_secret_cascade.sql` replaces the foreign key (`DROP CONSTRAINT IF EXISTS`, then `ADD CONSTRAINT`). - `server/src/services/secrets.ts`: `remove` accepts an optional `beforeDelete` hook. The hook runs after the secret is marked deleted and before its row is removed. - `server/src/services/secret-proposals.ts`: new `rejectPendingBindingsForSecret`. It rejects pending binding proposals for a secret with a reason and returns them. This closes their review cards and writes the normal activity entries. `createBinding` now holds the secret row `FOR SHARE` in its insert transaction and refuses a deleted secret. So no new pending proposal can start after the delete marks the secret deleted. - `server/src/routes/secrets.ts`: the delete route rejects pending proposals in `beforeDelete`. For each one it sends the normal rejection notice and wake (`notifySecretProposalResolution`), so the proposing agent continues. - `server/src/__tests__/secret-proposals-routes.test.ts`: a regression test deletes a secret with a pending, rejected and approved binding proposal through the route. It checks the secret and proposal are gone, the card status is correct, and only the pending case sends one rejection notice and one wake. A second test checks that a binding proposal for a secret that the delete already marked deleted fails with 404. ## Verification - Red first: the new test failed on `origin/master` with the shape-check violation for all three cases. - `cd server && vitest run --config vitest.config.ts src/__tests__/secret-proposals-routes.test.ts`: 47 passed. - `vitest run src/__tests__/chat-run-milestone-governed-work.test.ts src/__tests__/agent-secrets-routes.test.ts src/__tests__/secrets-service.test.ts`: 112 passed. - `tsc --noEmit` for `server` and `packages/db`: pass. `pnpm --filter @paperclipai/db run check:migrations`: pass. - Found by the `secrets.secret-proposal` chat evaluation. Its cleanup deletes the fixture secret and failed on this error in runs 42563093640 and 42571127911. ## Risks - The migration only replaces one foreign key. It does not rewrite rows. Existing rows already have a non-null `secret_id` or `secret_proposal_id`, so nothing else changes. - Deleting a secret now also deletes its binding proposal rows. The activity log keeps the proposal history (`secret.proposal.*` and `secret.deleted`). Secret-kind proposal rows that created the secret keep `created_secret_id ON DELETE SET NULL`, so their history stays. - Other paths that delete secrets (company deletion, user secret cleanup) now cascade instead of failing on the same constraint. They do not reject pending cards first. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d7d0f96f3f |
perf(runner): save provider state once per poll instead of once per frame (#20217)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode, Codex and Cursor Cloud chat turns run through runnerd. runnerd reads provider frames in order and keeps the run's provider state in a durable file > - For every frame that it reads, runnerd rewrote the whole provider state file and called `fsync` on the file and on its directory. The state holds every tool result kept in the turn, so each save got more expensive as the turn went on > - A tool call request is just the next frame. It waited behind one full save for every frame before it. #20209 measured tool-call turnaround that grew from about 1.4 s to about 15 s in one turn > - #20209 stopped the OpenCode proxy from sending frames that runnerd discards. Every frame that runnerd still reads still paid a full save, and other proxies (for example Cursor Cloud) still send such frames > - This pull request makes runnerd save the frames of one poll once, before anything derived from them leaves runnerd. A save that would write the bytes already on disk is skipped > - The benefit is that the number of saves before a tool call no longer grows with the number of streamed frames. In a local measurement, a tool call after 400 frames with 830 KB of kept state took 0.57 s instead of more than 30 s ## Linked Issues or Issue Description Refs #20209 Related: #17514 (the same per-event full-state rewrite, on the server control-plane side) **What happened** During a native OpenCode chat turn, tool-call turnaround grew from about 1.4 s to about 15 s as the turn went on. runnerd saved its complete durable provider state, with `fsync`, after every provider frame. That state includes the tool results kept so far in the turn. A tool call request waited behind one full save for each earlier frame. **Expected behavior** The time to deliver a tool call must not grow with the number of assistant-text, reasoning or item frames that came before it. A crash or restart must still never lose state that a resumed run needs. **Steps to reproduce** 1. Start a native OpenCode (or Codex app-server or Cursor Cloud) turn that keeps a few tool results and then streams many text or reasoning frames before its next tool call. 2. Measure the time from the tool call frame to the `semantic_tool.input` event. 3. See that the time grows with the number of earlier frames and with the size of the kept state. Locally, `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture` on `master` reports 102 saves and 9.3 s for 100 frames with 830 KB of kept state, and more than 30 s for 400 frames. ## What Changed - `packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs` and `codex_provider.rs`: - The provider frame loop moves into `poll_provider_frames`. An ordinary notification frame and a descendant notification frame no longer save the state. They mark it as unsaved. - After the loop, `poll_current_provider_with_control` saves the unsaved state once. It also saves when a frame fails, before it returns the error. - Frames that already saved at once still do so: turn terminals, Cursor Cloud terminal deliveries, `reconciliation_required`, protocol failures, MCP permission and runtime requests, process exit, and new tool calls. - A tool call that runnerd answers at once (an exact completed replay or the receipt-limit stop) saves unsaved state before it writes to the provider. A rejected tool call already saved first. - `retained_events`, `execute` and `host_park_ready` save unsaved state first, so no event, command result or park decision can expose state that is not on disk. - Courtesy rejections that `CodexProvider::poll` writes by itself (unsupported requests, descendant requests, delayed requests after a terminal, full runtime-request capacity) are held while runnerd reads a batch (`hold_courtesy_responses`). runnerd sends them only after it saves the state that covers every earlier frame (`save_before_provider_writes`). A terminal frame also saves and releases them before runnerd asks the provider for goal state or drains the accounting tail. Released courtesy replies are best-effort, as courtesy replies were before: a provider that already closed stdin cannot turn a received terminal into a polling failure. The receipt-limit stop sends held cancellations before its `turn/interrupt` request, so their order to the provider does not change. - The trace records for runnerd's normalization (`rust_durable_normalization`), which name emitted event ids, are kept until the state that holds those ids is saved. The trace therefore only names saved events. Saved mappings are written before an exited or failed provider is dropped, so they are not lost. Raw frame records still go to the trace when a frame is read; they name no event ids. - `save_unsaved_frame_state` does not write when the serialized state is byte-for-byte the same as the saved file. All other saves write as before. - Serialization moves to `serialize_provider_state`, and the file write moves to `write_state_bytes`. The write path (private temporary file, `fsync`, rename, directory `fsync`, `0600`) does not change. - New tests in `packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs`: - `streamed_frames_do_not_multiply_durable_saves_before_a_tool_call`: 8 and 48 streamed frame pairs before a tool call need the same number of saves (1). On `master` they need 18 and 98. - `batched_frame_state_is_saved_before_events_leave_and_survives_a_restart`: at every poll, the saved file is equal to the in-memory state and holds exactly the delivered events. A new executor loads the file after the old one stops and gets the same state, the same unacknowledged events and the pending tool call. - `batched_frame_state_is_saved_when_a_later_frame_fails`: a frame that fails after 20 batched frames still leaves those 20 frames on disk. - `previous_release_state_is_rewritten_by_the_first_batched_save`: a state file in an older shape (compact JSON without newer default fields) loads, and the first batched save rewrites it. Unchanged state is not rewritten. Changed state is. - `courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved`: when the courtesy rejection reaches the provider, the saved file (what a restarted runnerd would load) already holds the 20 earlier frames and equals the in-memory state. It fails if the hold is removed. - `trace_interpretations_name_only_saved_events`: every event id that a trace record names is in saved state at the moment the record is written. It fails if records are written before the save. - `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`: after 122 frames are read and not yet saved, the saved file is byte-for-byte unchanged and runnerd has written nothing to the provider. A reload gets the pre-batch event sequence and no pending events. - `a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`: the provider closes stdin and then completes the turn with a tool call still pending. The held cancellation cannot be written, and the turn still completes and is saved. - `trace_interpretations_survive_a_provider_exit_in_the_same_poll`: the trace keeps the mappings for frames that arrived in the same poll as the provider exit. - `measure_streamed_frame_cost_before_a_tool_call` (ignored): prints saves and tool-call latency for different frame counts and kept-state sizes. ## Verification - `cargo test --locked -p paperclip-runner-core` (debug, macOS arm64): library 595 passed, 1 failed; all 35 integration test binaries passed (`--no-fail-fast`). The one library failure is `task_memory::tests::monitor_bounds_notices`. It also fails when run alone, takes about 9 minutes, and waits on file-watch timing in `task_memory.rs`, which this PR does not touch. In one earlier run, two ACPX park tests in `acpx_park_preparation` failed. The same binary also failed on unchanged `master` (with a different test), and it passed in the final run. - `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture`, debug build, macOS APFS (`sync_all` is `F_FULLFSYNC`). Each row is one turn: frames streamed, then one tool call. | Kept state | Frames | Saves (master) | Tool-call latency (master) | Saves (this PR) | Tool-call latency (this PR) | | --- | --- | --- | --- | --- | --- | | 4 KB | 100 | 102 | 946 ms | 1 | 18 ms | | 4 KB | 400 | 405 | 3,791 ms | 7 | 70 ms | | 211 KB | 100 | 102 | 3,207 ms | 1 | 37 ms | | 211 KB | 400 | 405 | 12,803 ms | 7 | 193 ms | | 833 KB | 100 | 102 | 9,251 ms | 1 | 93 ms | | 833 KB | 400 | about 405 | more than 30 s (test deadline) | 7 | 567 ms | With 400 frames, the remaining saves are one per poll of up to 128 frames and one per controller acknowledgement. - With the debug `paperclip-runnerd` and `fake-codex-app-server` binaries from this branch: `npx vitest run` on `src/live/runnerd-opencode-streaming-tools`, `runnerd-opencode-assigned-tools`, `runnerd-codex-transport`, `runnerd-codex-transport-settlement`, `runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`, `runnerd-saved-continuation`, `runnerd-detach-durability`, and `src/cli/opencode-proxy-output`, `opencode-proxy-events`, `opencode-app-server-proxy`: 11 files, 353 tests passed. ## Risks - Durability. The rule is: nothing derived from a frame leaves runnerd before the state that holds that frame is on disk. That covers delivered events, command results, park decisions, tool results and rejections, courtesy rejections, Cursor Cloud terminal acknowledgements, and trace records that name event ids. The tests above check each of these. - The unsaved window is at most one poll (128 frames). Every frame in it is undelivered: no event from it reached the controller, and runnerd wrote nothing to the provider while it was unsaved. `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged` proves this. A crash in the window therefore leaves the saved state and everything the controller and provider have seen exactly as a crash just before those frames were read. The resumed run starts from the same event sequence, so it cannot deliver an event id twice. Its normal recovery (thread resume and `thread/read` reconciliation) handles those frames in the same way as frames still in the provider pipe. On `master`, a crash could already lose the one frame being processed. - One provider write can still happen in the window: `thread/read`, which `CodexProvider::poll` sends to check the lineage of an unknown helper thread. It is a read-only query that waits for its answer, and it changes nothing on the provider side. - Courtesy rejections now reach the provider at the end of the poll instead of during it. The delay is one batch. The provider gets the same replies. - Previous-release story: the saved file shape does not change, and no new field is saved. A file from the previous release loads as before. The skip check compares exact bytes, so an older file is never treated as already saved. The first save after an upgrade rewrites it in the current shape. A deploy that lands mid-turn sees a file that the previous release saved after each frame. That file is valid input for this code. - The ACPX backend has its own frame loop and is not changed here. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file editing) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
088bdf7cfa |
fix(daytona): wait out a sandbox state change before starting it for workspace realize (#20212)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, every run realizes its workspace in the leased sandbox. Realizing it first makes sure the sandbox is started > - Right after a control-plane restart, the sandbox can still be mid-transition, for example stopping or starting from the previous lifecycle step > - The plugin called `sandbox.start` straight away. Daytona rejects that with "Sandbox state change in progress", so the run failed `setup_failed` and needed a Retry > - This pull request waits out a sandbox's state change before starting it, and retries a start the provider rejected for that reason, within the existing start deadline > - The benefit is that a run whose sandbox is briefly transitioning starts on its own ## Linked Issues or Issue Description Refs #20211 **What happened?** Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503 (revision |
||
|
DottaandClaude Opus 5.5
|
9c18cf61da |
fix(runner): keep discarded OpenCode token deltas from delaying tool calls (#20209)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode runs use a TypeScript app-server proxy between OpenCode and runnerd > - runnerd reads the proxy's frames in order, only after the controller acknowledges earlier events > - For each frame, runnerd rewrites its whole durable provider state with fsync, including every tool result that the turn retains > - The proxy sent one `item/delta` frame for each reasoning or tool-output token, and runnerd discards `item/delta` > - A Paperclip tool call request therefore waited behind hundreds of discarded frames, and the wait grew with each tool result in the turn > - This pull request stops the proxy from sending the token deltas that runnerd does not consume > - The benefit is that tool calls reach runnerd without that queue, so agent turns finish faster ## Linked Issues or Issue Description **What happened** In the chat eval `plugins.configure-plugin` (runner-opencode, deepseek-v4.1-flash, GitHub Actions run 42574640038), `paperclip_search_api` calls in one turn slowed from about 1.4 s to 9-15 s each. The server handled every call in 50-250 ms (`tool.execution` spans). For one `call_api` call, 6.3 s passed between the server's 403 and the next agent activity. Time to first token was about 4.4 s. The run events show the cause. Events that OpenCode streams reach runnerd in bursts. In one burst, the usage events for three different model requests arrived within 0.5 s, which no model can do. Each burst ends with the next tool call request. 143 of 203 recorded runner-opencode runs that we checked show such bursts, and 108 show a gap of 8 s or more. Many chat eval cases that time out use this runner. The mechanism: 1. OpenCode streams reasoning one token at a time. The proxy forwarded each token as its own `item/delta` frame. 2. runnerd reads proxy frames in order and only after the controller acknowledges earlier events. For each frame it calls `save_state`. That serializes the whole provider state, with the tool results retained in the turn, and fsyncs it. 3. runnerd's Codex facade has no mapping for `item/delta`, so it discards the frame after the save. 4. A tool call request is one more frame in the same ordered stream. It waits until runnerd works through every earlier delta frame. The wait is about (reasoning tokens) x (save cost), and the save cost grows with each retained tool result. **Expected behavior** A Paperclip tool call reaches runnerd soon after OpenCode makes it. Its delay does not grow with the reasoning length or with the number of earlier tool calls in the turn. **Steps to reproduce** 1. Run a native OpenCode chat turn with a reasoning model that makes many sequential Paperclip tool calls, for example `search_api`. 2. Compare each tool's `item.completed` tool_result time with the `tool.execution.completed` time and with the next `item.started` tool_use time in the run events. 3. See that the gap grows during the turn, and that events for several model requests arrive within less than one second. 4. For a local model of the mechanism: run the pinned OpenCode binary with a controlled OpenAI-compatible provider that streams 300 reasoning tokens before each tool call. Pass the driver events through the proxy's mapping and batching into a reader that consumes one frame at a time with a save cost that grows by 4 ms per completed tool result. Before this change, 8 sequential calls waited 1.7, 3.1, 4.2, 5.4, 6.7, 7.9, 9.2 and 10.5 s behind 142-261 queued frames. After this change, they waited 64-93 ms behind 0 frames. ## What Changed - `shouldForwardOpenCodeProxyItem` now takes the event type and payload. For `item.delta`, it forwards only assistant text (`agentMessage` or `text`), which runnerd maps to `item/agentMessage/delta`. - The proxy keeps reasoning and tool-output token deltas. Completed reasoning and tool items still go to runnerd as `item/completed`. Tool progress still goes as `item/updated`. - New regression test: a step with 1,000 streamed reasoning and tool-output tokens puts the same number of frames before the tool call request as a step with 1 token, and no `item/delta` frame is written. - New test: assistant text deltas are still forwarded. Completed reasoning items are still forwarded. ## Verification - `pnpm exec vitest run src/cli/opencode-proxy-events.test.ts src/cli/opencode-proxy-output.test.ts` in `packages/paperclip-runner`: 15 passed. - Red/green: with the predicate change reverted, the two new tests fail. With the change, they pass. - `tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner` has no errors in the changed files. Local errors exist only in `src/eval/workflow-harness.ts`, which needs a built `@paperclipai/paperclip-eval-kernel`. - Local before/after measurement with the real pinned OpenCode binary (`opencode-ai@1.18.34`), described in Steps to reproduce, step 4. - Not run locally: the suites that need the `paperclip-runnerd` debug binary or the bundled provider entrypoints (`runnerd-opencode-streaming-tools`, `runnerd-codex-transport`, `opencode-proxy-thread-read`). They fail the same way on the unchanged base in this worktree because those build outputs are missing. CI covers them. ## Risks - Low risk. The changed frames were discarded by runnerd. No persisted state shape changes, so no previous-release story is needed. - If runnerd later maps a reasoning delta method, the proxy must forward it with batching, as it does for assistant text. The test names this boundary. - This change does not change the per-frame durable save in runnerd. Mapped frames (assistant text, tool items) still pay it. A runnerd change that skips the save when a notification changes no state would protect other proxies too. That is a separate change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, local benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
42bc238aa5 |
fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records it as `provider_turn_lost_on_restore` and continues in a same-conversation continuation turn > - #20118 taught the event consumer to skip a stale restart-lost terminal of another turn once a continuation is running. It applied only when the checkpoint already recorded the interruption > - When the checkpoint instead names the lost turn as active (the drain hit right after the first turn was submitted), the consumer starts the continuation on the first lost terminal. If the same lost terminal is then delivered again, it was taken as the run's terminal and failed the run > - This pull request applies the skip to any recovered run once its continuation has started. For a checkpointed interruption it still waits for the startup decision first > - The benefit is that a turn lost on restore always continues, however often its lost terminal is delivered ## Linked Issues or Issue Description Refs #20118 **What happened?** Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master |
||
|
Nicky LeachandClaude Opus 5.5
|
9046884811 |
perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner executes native agent runs. A host can park an idle
Runner (release it and freeze it) between runs, and resume it for the
next run on the same task.
> - A follow-up run that reuses a parked Runner must claim it first. The
claim is a bootstrap ticket, a checkpoint flush, and the plugin resume
that thaws the Runner.
> - Core made the claim only inside `attachRun`, after all of its
per-run preparation. The claim does not depend on that preparation, so
the two ran in series.
> - This pull request starts the claim while Core still prepares the
run. `attachRun` then waits for the claim that is already in progress.
> - The benefit is a shorter start for every follow-up run on a parked
Runner. The model below gives about 285 ms per run at P50.
## Linked Issues or Issue Description
No public issue exists. The issue is described inline.
**What happened?**
A follow-up run that reuses a parked Runner claims the Runner only in
`attachRun`. Core calls `attachRun` after backend setup: runner
environment, assigned MCP tools, tool authority and the plugin task
owner. The claim waits for all of this, but it uses none of it.
**Expected behavior**
The claim runs at the same time as backend setup. Everything that needs
the new run's tool authority stays at attach.
**Steps to reproduce**
1. Run a Codex or Claude task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. Before this change, the plugin resume starts only
after backend setup ends.
**Paperclip version or commit**
`master` at
|
||
|
Nicky LeachandClaude Opus 5.5
|
92a18bb0f3 |
perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner drives Claude through the ACPX sidecar. The sidecar's MCP endpoint lasts for the whole session, so it cannot tell which turn sent a late tool callback. > - Because of this, the Runner reaps the sidecar and provider of a generation that ran a turn before it releases call-ID tombstones and rotates receipt epochs. Today this restart runs at the start of the next turn. > - Every follow-up turn in a run pays that restart on its critical path: about 1.5 s with real Claude. #20145 moves the restart to park, but a second turn in the same run, after the first turn that follows an attach, still restarts cold. > - The restart does not have to wait for the next turn. When a turn settles and the host has its terminal result, the Runner is idle. > - This pull request restarts the provider at that point. The next turn, or the next park, then finds a fresh generation. > - The benefit: same-run follow-up turns start without a restart, park does no restart after a settled turn, and the late-callback guarantee is unchanged. ## Linked Issues or Issue Description Refs #20145, which moved the restart to park. This pull request builds on it. No public issue exists, so the problem is described here. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** `AcpxProviderSession::start_turn_timed` restarts the provider before any turn whose generation already ran a turn. A follow-up turn in the same run waits for `session.suspend`, sidecar shutdown, a new sidecar, `initialize`, `session.open` and `run.attach` before `turn.start`. Since #20145, park does the restart instead, but only for the first turn after an attach. **Proposed behavior** When a turn settles and the controller has ACKed every committed event, the Runner restarts the provider. The next `turn.start` uses the fresh generation and only rotates receipts. Park reuses the fresh generation and does no second restart. **Reason and benefit** The restart leaves the turn's critical path, both for same-run follow-ups and for the turns after a park. The order of reap and rotation stays the same, so a late callback from the reaped generation still cannot reach a later turn. ## What Changed - **Runner loop** (`durable/runner.rs`): new `CommandExecutor::prepare_next_turn` hook, a no-op by default. The runner calls it after the idle provider poll, only when no control frame is waiting and the controller has ACKed every committed event (`acked_source_seq >= highest_source_seq`). So the restart always runs after the terminal result reaches the host. - **ACPX turn-end restart** (`acpx_park_preparation.rs`, `acpx_provider_generation.rs`): - `restart_session_after_turn` restarts a Claude session that has a completed turn, nothing pending, no buffered events, a dormant goal and its whole event tail ACKed. It first settles an abandoned park observation. If that settlement reads more events, it skips this round, and the next idle round restarts after those events are ACKed. - `AcpxProviderSession::restart_after_turn` does the same suspend → reap → start → bootstrap → identity check as the turn-start restart. It records the reaped process id in `restarted_after_turn_from`. This field is the explicit "fresh because of a turn-end restart" marker. - The session does not poll a fresh generation until its next turn or rebind, the same as a prepared one. Its startup output is published with the next turn, as today. Polling it early would add events after the host computed its park cut. - **Next turn** (`acpx_provider_session.rs`): `start_turn_timed` restarts only when the generation ran a turn. When `restarted_after_turn_from` is set, it does the post-restart receipt and ledger rotation without a second restart. `session.reconciled` reports the reaped process as `previousProcessId`. - **Park**: `prepare_next_generation` restarts (with the 20 s park deadline) only if no turn-end restart ran. Otherwise it marks the generation for `run.attach` rebind and returns `false`, so the runner does not ask for a second park proof. The park-time receipt rotation moves to the next turn start, where the turn-end path does it too. - **Restart failure**: the executor sets the same state that a failed turn-start restart leaves (`closed`, no session, provider exit confirmed). The next `turn.start` fails with the same `ACPX turn start failed closed: …` error that the turn-start restart gives today. - **Fake sidecar**: new `pending-input` fault, which opens an input request in a turn that then completes. ### Races | Race | Handling | | --- | --- | | A turn or steer arrives during the turn-end restart | The runner loop is single-threaded. The command waits in the socket until the restart returns, then starts on the fresh generation without a restart. Only one generation is live. | | A turn arrives before the hook runs (for example, in the same read as the ACK) | Control frames go first, so the turn restarts at its start, as today. The hook then restarts after that turn. | | Park arrives during the turn-end restart | Host control is read at the top of the loop, so park waits. Park then reuses the fresh generation. No event is published in between, so the host's park cut still matches. | | Park arrives before the hook runs | Park restarts with the 20 s deadline, as in #20145. The hook then finds a fresh generation and does nothing. | | Eviction, `runner.suspend` or retire after the restart | They reap the fresh generation through the ordinary paths. | | The restart fails | The session is `closed`. The next turn fails with the same error and the same durable state as a failed turn-start restart. Park returns conflict, as it does for a closed session today. | | Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after the restart return `already_settled`, as after any settled turn. A cancellation that retires the runtime closes the session, so no restart runs. | | Interrupted, cancelled or failed turn | `completed_turn_observed` is false, so no turn-end restart runs. The next turn restarts at its start, as today. | | Pending permission or input requests at the terminal | The reducer ends them at the terminal, so the restart runs. A later `request.resolve` fails with `no active turn`, with or without the restart. | | Active session goal at the terminal | No restart, because the goal can keep working after its prompt. | ### Removed as redundant, and what stays - **Removed:** the park-time restart and dormant verify when a turn-end restart already made the generation fresh. The runner's `host_park_ready` proves dormancy just before `prepare_for_park`, and nothing runs in between. - **Removed:** the runner's second park proof in that case, because the provider did not change. - **Removed:** the park-time receipt and ledger rotation. The next turn start does it once, for both paths. - **Kept: the turn-start restart.** Without it, a turn queued before the hook, a turn after an interrupted or cancelled turn, a turn after an active goal, and any non-Claude ACPX turn would run in the generation that ran the previous turn. Tests cover the queued-turn and interrupted cases. - **Kept: the park-time restart.** Without it, a park that arrives before the hook would leave the next run to restart at attach. A test covers this. - **Kept: the settlement of an abandoned park observation.** Without it, the restart drops events that the old generation sent before the abandoned response. A test covers this, and it fails with the settlement removed. - **Kept: the goal-dormancy check.** Without it, the restart kills an active goal's autonomous work. A test covers this, and it fails with the check removed. - **Kept: the ACK gate.** Without it, the restart can run before the host has the terminal result. A runner test covers this. - **Kept: deferred polling of the fresh generation.** Without it, its startup event lands after the host computed its park cut, and the park conflicts. - **Kept: the kept failure message.** Without it, the next turn fails with `ACPX session is not open`, and the restart error is lost. ## Verification - `cargo test -p paperclip-runner-core --lib` and every `tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery` suite: all pass (debug). - New tests (`tests/support/acpx_turn_end_restart.rs`, run from `acpx_park_preparation`) prove these points: - A settled turn reaps the old pid before it spawns the new one, and no predecessor is still live. A same-run follow-up then sends only `turn.start` and reports the reaped pid in `session.reconciled`. - Park after a turn-end restart sends no sidecar command and returns `false`. Attach rebinds the same pid. The run's second turn restarts at its end, and the third turn starts without a restart. - The reaped generation's late callback never reaches the next turn. - A turn that arrives before the hook restarts at its start. A park before the hook restarts once. - A failed turn-end restart gives the same next-turn error, retry error, durable state and process count as a failed turn-start restart. - Cancellation after the restart is `already_settled`. `runner.suspend` reaps the fresh generation. - An active goal skips the restart. An input request open at the terminal behaves the same with or without the restart. - `acpx_provider_turns`: a session-level turn-end restart drops the old generation's late `issues.delete` callback, which reuses the previous turn's call ID. The next turn then admits that call ID from the fresh provider. Interrupted and retired turns do not restart at turn end. - `acpx_park_readiness`: a turn-end-restarted generation is park-ready. An abandoned park observation tail is delivered before the restart. - `durable::runner`: the hook runs only after the terminal is ACKed and when control is idle. It never runs during ACK debt, a waiting control frame or a disconnect. - Server integration tests (`plugin-runner-release-resume`, `plugin-runner-eviction` and `plugin-runner-task-warm`) with this branch's Runner: 49 passed. The provider process counts do not change: each released run still adds one process start, now at turn end instead of at park. - `cargo fmt --check`: clean. - Fake-sidecar timing (release build, 7 runs, medians): | | before (`master` with #20145) | this PR | | --- | --- | --- | | same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms | | second turn after an attach, `turn.start` → `turn.started` | 270.3 ms | 23.5 ms | | turn-end restart, after the terminal is ACKed | — | 29.0 ms | | park preparation after a settled turn | 34.1 ms | 0.03 ms | - Real Claude: not run for this pull request. With real Claude, the restart took about 1.5 s on the turn-start path, and this change takes it off that path. That number is not measured on this branch. ## Risks - A run that ends after one turn without a park now starts one extra sidecar that no turn uses. The Claude CLI itself starts only at the first prompt, so the cost is the sidecar's start and bootstrap. - The restart blocks the Runner loop for its length (about 1.5 s with real Claude), as the turn-start restart does today. It has no deadline, as before. A park that arrives during it waits, inside the host's 30 s park wait. - The fresh generation is not polled until its next turn. If it exits while idle, the next turn finds out at `turn.start` and fails closed. Park finds out at its observation. - The turn-end restart is for Claude only, the same as park preparation. Other ACPX agents keep restarting at turn start. - If the Runner crashes during the restart, the durable state still says `session_open`. Recovery is the same as for a crash while idle. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
2d93413085 |
perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path
> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.
## Linked Issues or Issue Description
No public issue exists. The issue is described inline.
### What happened?
On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.
### Expected behavior
The respawn happens while the Runner parks, so the next run starts on a
prepared generation.
### Steps to reproduce
1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.
### Paperclip version or commit
`master` at
|
||
|
DottaandClaude Opus 5.5
|
5a51a7685d |
fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop cancels the scope's running work; the board resolves it with Raise budget and resume > - In the restart-chaos H1 scenario, the cancelled tasks did not continue after resume and each needed a manual Retry > - Resume is the board's explicit decision to continue, so the work it stopped should continue once, as if the board clicked Retry > - This pull request files that Retry automatically, keyed by the cancelled run, once the run's stop has settled > - The benefit is that a budget raise restores the work without manual Retries, without repeating completed work, and without charging the run's attempt budget ## Linked Issues or Issue Description Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101 (superseded: split into #20143 and this PR) **What happened?** Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802 (H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run (enforcement stop) and CHA-1's run (agent lifecycle pause from the budget hold) stayed cancelled until someone clicked Retry. **Expected behavior** Resuming continues each task its budget stopped, exactly once, with no attempt charged, an activity entry, and no repeat of work that already continued. A run a person stopped stays stopped. **Steps to reproduce** Set a budget below spend while a task runs; let the stop cancel the run; resolve with Raise budget and resume. Before this change, no new run is queued. ## What Changed - New table `budget_resume_claims` (migration 0451). One row per cancelled run, outside the run's result JSON, so a Stop that settles during the resume cannot erase it. - `markBudgetCancelledWorkForResume` (called by `resolveIncident` before the incident closes): claims every task run its budget stopped since the incident opened. That covers runs budget enforcement fenced, including stops still in flight, and runs the agent lifecycle pause stopped for this budget (`pauseStop`, carried from the lifecycle operation's `pauseCause` so a resume before the Stop keeps the link). Runs a person stopped are excluded. Two indexed passes led by `(company_id, status)`. - `deliverBudgetResumeContinuations`: once a claimed run's stop settles, files the resolving board member's Retry (`requestTaskRetry`, idempotency key `budget-resume:<run>`, with the member's session source and the chat Retry callback). The Retry path checks, under the task lock, that the run is still the task's latest and the task is open, so continued or completed work is never repeated. Claim settlement and its activity entry commit together; retryable refusals and errors retry. - Delivery runs in the always-on budget-enforcement worker after each pass's Stop delivery. Its schedule includes deliverable claims and the earliest stale-claim expiry. The execution-continuation worker is a backstop. - `agent.paused` / `agent.resumed` logging is in #20113. `budget.cancelled_work_resume_requested` (incident) and `budget.cancelled_work_continued` / `_not_continued` (task) are recorded here. ## Verification - New `budget-resume-continuation.test.ts` (Postgres, real `resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry` into a queued run): - each task continues once, including a blocked task; repeat delivery or resolve saves nothing new; - a stop still in flight at resume, whose terminal write replaces the result JSON, is still continued; - a 30-task scope is fully continued; - a lifecycle-pause stop caused by the budget is continued, and one caused by a manual pause is not; - continued work, done tasks, a run that finished before its stop, and person-stopped runs (including ones enforcement also fenced) are not continued. - `lifecycle.test.ts`: the pause records its cause on the operation and on stopped runs, including when the budget is resumed before the pause's Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery, and a wake at claim expiry. - 287 tests across the affected suites passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (Phase 3 runs it after merge). ## Risks - Resume now continues budget-stopped tasks automatically. A board that wants a task stopped can Stop it again; person-stopped runs are never continued. - A new table and one index (migration 0451). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b54317d3fb |
fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.
## Linked Issues or Issue Description
**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.
**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.
**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.
**Paperclip version or commit**
Staging revision
|
||
|
DottaandClaude Opus 5.5
|
28e070f8dc |
feat(skills): show a line diff for a pending tracked-source update (#20155)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Skills can come from a tracked GitHub source, and an update moves
the installed copy to a newer upstream revision
> - Before an update, a person asks the agent "what changed in the new
version?"
> - The product has no diff for a pending update. The agent previews two
full SKILL.md snapshots and compares them by eye
> - In two eval runs the agent reversed the same changed line. In a
third run it fetched the files with curl and ran `diff` itself, and the
turn took too long
> - This pull request adds a read-only endpoint that returns the line
diff between the installed and the latest tracked revision
> - The benefit is that review answers come from an exact diff, and they
come faster
## Linked Issues or Issue Description
No public issue exists. Feature description:
**What happened**
An agent asked to show what changed in a pending skill update had only
`POST /skill-sources/preview`, which returns one file at one commit. It
previewed both commits and compared about 9 KB of text by eye. It
reported one changed line in the wrong direction ("every new build" ->
"every build or release" instead of the reverse), and later retracted a
real change.
**Expected behavior**
The agent can request the exact changes of a pending tracked-source
update and report them with the correct direction.
**Steps to reproduce**
1. Install a skill from a GitHub source at an old commit and set the
tracking reference to a newer commit (automatic updates off).
2. Ask the chat agent: "Show me what changed in the new accessibility
skill version before we update it."
3. See that the agent can only preview full files at each commit and
must compare them itself.
## What Changed
- Add `GET /api/companies/{companyId}/skills/{skillId}/update-diff`. For
a skill from a tracked GitHub source, it compares the installed source
revision (`currentRef`) with the latest tracked revision (`latestRef`).
For each changed package file it returns `change` (added, removed,
modified, or `not_compared`), `binary`, `executableChanged`, and a
unified line diff (`-` installed, `+` latest), or `diffUnavailable`
(`binary`, `too_large`, `omitted`, `budget`) that says why there is no
diff. The endpoint is read-only. The `skillId` parameter handler checks
company access and skill visibility, the same checks other skill reads
use.
- `hasUpdate` is false and `files` is empty when the installed revision
is already the latest one. That case reads only the latest commit and
scans nothing.
- Return `supported: false` with a reason for: a skill without a tracked
source, a disconnected source, an unknown installed revision, and a
repository package that a scan could include only in part (a subset, or
skill folders only). In the last case, files that a scan limit left out
would otherwise look removed.
- A path the scan left out of a revision (too large, a symlink, or
rejected by the audit) is listed as `not_compared` instead of added or
removed.
- One response shares a work budget (24M LCS cells), a diff-text budget
(1M characters) and a file-count budget (1,000 files), and sets
`truncated` when it reaches one. The GitHub read and both scans stop
when the caller disconnects.
- Add `server/src/services/skill-update-diff.ts` with a bounded line
diff (shared head and tail trimmed, LCS size capped). An absent side has
no lines, a final newline is not a line, and a missing final newline
shows the unified-diff marker. A replaced line reads `-old` then `+new`.
- Add the shared types `SkillSourceUpdateDiff` and
`SkillSourceUpdateDiffFile`.
- Document the endpoint in the API catalog (`openapi.ts`) so agents find
it with `search_api`.
## Verification
- `cd server && npx vitest run src/__tests__/skill-update-diff.test.ts`
(10 tests):
- direction of `-`/`+` lines, and separate hunks with bounded context
- exact diffs for added and removed files, and final-newline handling
- a change to only the executable bit, and paths left out of a scan
- the response budgets
- the snapshot comparison for the selected skill path, with the abort
signal passed to both scans
- a commit mismatch
- refusal of partial repository scans
- `cd server && npx vitest run
src/__tests__/skill-update-diff-routes.test.ts` (6 tests):
- the diff response
- no pending update (no scan)
- a caller from another company (board user and agent) gets 403 before
any lookup
- a skill the caller cannot read gets 404 before any source read
- every unsupported case
- a client disconnect aborts the scans
- `company-skills-routes.test.ts` and `openapi-routes.test.ts` pass (119
tests across the four files).
- `npx tsc --noEmit -p server`: no errors from this change.
- End to end: the `skills.review-update` chat eval exercises this path.
Its grader checks the quoted hunks against the real pinned diff.
## Risks
- Low risk. The endpoint is new and read-only. Existing routes do not
change.
- When an update is pending, each call scans the source twice (installed
and latest commit). It uses the same GitHub reader, scan limits and scan
slots as preview and refresh. The diff work and response size are capped
per response.
- `/update-status` and this route both read the latest commit of the
tracking ref. They are kept separate because `/update-status` compares
with the source's last scan, while this route compares with the
installed revision.
- No UI uses the endpoint yet.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
cedf335c98 |
fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records that turn as failed with `provider_turn_lost_on_restore` and continues on a same-conversation continuation turn > - Restart recovery blocks a run as `provider_checkpoint_permanently_failed` when its checkpoint carries a failed run terminal and no pending restart interruption > - While a continuation is running, the checkpoint keeps the lost turn's failed run terminal and names the continuation as the active turn. The pending-interruption check returns nothing once a turn is active > - So a second restart during the continuation read the run as permanently failed, blocked replay, and needed the board > - This pull request treats that failed terminal as the restart loss it records whenever an active turn follows a restart-lost terminal turn > - The benefit is that runs survive repeated restarts during their continuations, including M1 parents and their subtasks ## Linked Issues or Issue Description Refs #20148 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master |
||
|
Devin FoleyandPaperclip
|
546cfee919 |
fix(github): send gh repo create and fork to the connection's tools (#20142)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents with a managed GitHub connection get `git` and `gh` launchers that use the connection's credential > - The launcher gives that credential to any `gh` command, and the broker does not see which command runs > - So `gh repo create` and `gh repo fork` skip the connection's tool switches, ask-first approvals, owner defaults, and audit > - The connection's `create_repository` and `fork_repository` tools apply all of these > - This pull request stops `gh repo create`, `gh repo new`, and `gh repo fork` when a managed credential is in use, and names the tool to use instead > - The benefit is that repository creation follows the connection's rules, whichever path the agent tries first ## Linked Issues or Issue Description Refs #20141 (default owner for `create_repository` and `fork_repository`). **What happened?** An agent with a managed GitHub connection can run `gh repo create` in its shell. The command uses the connection's token, but it does not use the connection's tool rules. If the user turns off or reviews `create_repository` on the connection, the agent can still create a repository with `gh`. The owner default from #20141 also does not apply, so the command can fail with `403 Resource not accessible by integration` when the App is installed only on an organization. **Expected behavior** With a managed connection, repository creation and forks go through the connection's tools. The tool switch, approvals, owner default, and audit apply. **Steps to reproduce** 1. Connect GitHub with a managed connection. 2. Set the connection's `create_repository` tool to off or ask first. 3. In an agent run, run `gh repo create demo --private`. 4. The command runs with the connection's credential and ignores the tool setting. ## What Changed - `packages/adapter-utils/src/github-launcher.ts`: when the broker returns a managed credential, `gh repo create`, `gh repo new`, and `gh repo fork` exit with code 1 before `gh` starts. The message names the `create_repository` or `fork_repository` tool and tells the agent to finish with Git. - `--help` still runs. Other `gh` commands do not change. Executions without a managed credential run these commands as before. - `doc/execution-github-identity.md`: describes the behavior and states that this is guidance, not a security boundary. ## Verification - New test in `packages/adapter-utils/src/github-launcher.test.ts`: "sends repository creation and forks to the connection's tools when a connection is in use". It covers `create`, the `new` alias, `fork`, `--help`, another `gh repo` command, and the case without a connection. It fails without the launcher change. - `npx vitest run src/execution-target.test.ts src/github-launcher-environment.test.ts src/github-launcher.test.ts` in `packages/adapter-utils`: 83 passed. - `npx vitest run src/drivers/acpx/github-launcher.test.ts` in `packages/paperclip-runner`: 3 passed. - `npx vitest run src/services/heartbeat-github-launchers.test.ts` in `server`: 12 passed. - `npx tsc --noEmit -p packages/adapter-utils` passes. ## Risks - Agents that used `gh repo create --source . --push` must now call `create_repository`, then add the remote and push with Git. The message tells them to do this. - If the connection's tool is off for the agent, the agent cannot create a repository. This is the purpose of the tool setting. - This is not a security boundary. Other commands that receive the token, such as `gh api`, can still call the same GitHub endpoints. - Running launchers keep their earlier source until the next run stages new launchers. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c49ab6b9ee |
feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task delegates to children and waits on blockers. Its agent learns of progress only through wakes > - Today only completion (`issue_children_completed`) and resolution (`issue_blockers_resolved`) wake a waiting issue > - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on a recovery action that needs the board, and its parent sat silently blocked for 50+ minutes > - This pull request wakes the waiting agent once per such stall, tells it what it can do, and shows the stall on the parent's page with the child's Retry > - The benefit is that a stuck child or blocker no longer silently strands its parent ## Linked Issues or Issue Description Refs #19831 (durable ownership plan; restart chaos campaign). Restart chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude). **Problem** From the kept database: - CHA-8 `blocks` CHA-5. - CHA-8's board-owned recovery action (`native_session_retry_exhausted`) was created at 18:44:02.909 and resolved with outcome `blocked` at 18:44:08, when CHA-8 moved to `blocked` for the board. - CHA-5's only wake in the whole wave was its `issue_assigned` at 18:20:48. Its agent never heard about CHA-8. **Expected behavior** - When a child or blocker stalls on the board, the waiting agent is woken once. It can then re-delegate, escalate, or block with a reason. - The board sees the stall on the waiting task, with the child's Retry. ## What Changed - **Shared:** - `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`), `ISSUE_BLOCKER_STALL_RELATIONS`, and the types `IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`; - `Issue.stalledDependencies`. - **Server, `services/recovery/blocker-stall-wakeups.ts`:** - A stall is an issue that is not done or cancelled, whose latest recovery action needs the board. That means either open (`active`/`escalated`) and board-owned or escalated, or board-owned and resolved as `blocked` while the issue is still `blocked`. The second case is what t05 needed: recovery resolves its action that way when it hands the issue to the board. - Waiters are the parent, and every issue the stalled one `blocks`, when they are open, visible and agent-assigned. - Each waiter gets one wake per stall. The idempotency key `issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked through the indexed `(company_id, payload->>'issueId')` path. A later stall has a new action id, so it wakes again. - Each wake writes activity `issue.blocker_stalled_wake`. - The pass runs from the recovery-action sweep (`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep (`reconcileResolvedDependencyWakeBackstop`, which visits each blocked issue). No new worker. - A waiter's agent that cannot read the stalled issue (`issue:read`, which honors private visibility) is woken without its title, identifier or recovery details; the activity entry withholds them too. - **Database:** migration `0450` adds the partial unique index `agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id, idempotency_key)` for `issue_blocker_stalled:` keys in delivered or pending statuses, so two concurrent sweeps cannot both deliver a stall. It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`). - **Queue:** `allowsDependencyBlockedWake` (in `run-dispatch/domain/wake-context.ts`) lets a stall wake run while the waiter's blockers are unresolved, at both dependency gates. Without it the wake was parked until the stalled blocker resolved, which is never. Pause holds are unchanged. - **Agent guidance:** the wake payload carries `blockerStall`, and `renderPaperclipWakePrompt` adds a "Stalled dependency" section. It names the task and its recovery action and the board's next action, then the three choices: re-delegate, escalate with a comment, or block with a reason naming the stalled task. - **Other reason lists:** `issue_blocker_stalled` joins the responsible-user wake reasons, the conversation execution-wake reasons (it never drives a chat turn) and the wake diagnostic's known reasons. - **Board surface:** - `GET /issues/:id` returns `stalledDependencies`, readable issues only (`readableIssueEdgeIds`). - `StalledDependencyNotice` shows on the issue's chat notices, next to `IssueBlockedNotice`: each stalled child or blocker, its board next action, a link, and the child's own Retry (`useTaskRetry`), offered only when the child's interruption allows it. ## Verification Red/green: - `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests). - On master: no `issue_blocker_stalled` wake is raised. - With this PR: the parent is woken once per stall, and re-woken after a new stall. A blocker's dependent is woken. - The t05 shape (an action resolved as `blocked`) wakes once, and nothing more after a Retry. - No wake for an agent-owned action, a done, unassigned or hidden waiter, or an action resolved another way. - `listStalledDependencies` lists children and blockers. - `heartbeat-dependency-scheduling.test.ts` "admits an issue_blocker_stalled wake while a blocker is unresolved". On master the wake was parked (`null`); once admitted, the run-start gate cancelled it. With this PR it runs with `dependencyBlockedInteraction`. The file: 14 passed. - `packages/adapter-utils` `server-utils.test.ts`: the new prompt test failed on master (no `blockerStall`). The file: 141 passed. - `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed. `IssueChatThread.test.tsx`: 109 passed. Regression and static checks: - Nearby server suites (recovery durable wait, dependency wake routes, recovery actions, run-preparation): 149 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. - `pnpm check:token-gates` is clean. Review round 1 (Greptile, Superagent), each with a regression: - A backlog waiter is never woken. The waiter's status and assignee are guarded at enqueue. - One stall per run: no coalescing, so a merged run cannot keep only the last stall's context. - Only a delivered or pending wake ends a stall. A refused one (wake-on-demand off) is retried after 10 minutes, and an agent that takes no wakes is skipped without writing a row. - Company-wide passes resume after the last stall a capped pass reached. The per-issue sweep also checks the issue's own children and blockers, which covers an unassigned stalled blocker. - The dependency gates no longer trust the reason string. `verifyBlockerStallWake` requires a system request, the stalled issue's current stall action, and a real child or blocker edge. An agent-requested wake with the same context is parked (tested). - The notice shows a refused Retry's reason, and a lost response as "Taking longer than usual…", with Retry still available (tested). Review round 2 (Greptile), each with a regression: - Race: two passes claiming one stall leave one delivered row (unique index; the loser's insert is skipped). - The company recovery sweep selects companies whose only stall is a resolved-`blocked` board action, so an unassigned stalled blocker with a todo waiter is reached. - Private stalled issue: the waiter's wake and activity carry no title, identifier or recovery details. - `GET /issues/:id` always returns `stalledDependencies`, so a cleared stall replaces the cached list. - The notice shows a failed child read with a Reload button (tested). - `pnpm --filter @paperclipai/db check:migrations` passes. Review round 3 (Greptile), each with a regression: - A stall wake that fails to enqueue is retried: the recovery-action pass reports `blockerStallWakeFailures` and the dependency pass counts it in `enqueueFailed`, so the worker throws and retries at its backoff (`issue-recovery-work-lifecycle.test.ts`). - The notice drops a stall once the child's refreshed status has moved on (`StalledDependencyNotice.test.tsx`, 7 tests). ## Risks - A deploy wakes the waiters of every issue that is stalled today, once each. That is intended (they are stranded now), and it's bounded at 200 stalls per pass. - A stall wake may run while blockers are unresolved, like a comment wake. It is bounded to one per stall, and pause holds still apply. - The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet list the new reason. The wake prompt carries the guidance. - One additive index, built concurrently; no persisted run state change. Existing rows cannot violate it before deploy, since no `issue_blocker_stalled:` keys exist yet. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b43cccbb08 |
fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the runner reports it as `provider_turn_lost_on_restore` and the controller starts one same-conversation continuation. A later restart that loses the continuation is answered the same way > - With back-to-back restarts, the controller's checkpoint can miss a continuation that the runner already accepted. On the next restore the controller starts a fresh continuation, and the restored runner then reports the unrecorded earlier continuation as lost > - The event consumer treated that stale lost terminal as the run's terminal, so the run failed while its new continuation was already running > - Recovery then refused to replay the failed run, and the task needed a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but the parent stayed `blocked` on that subtask > - This pull request makes the consumer skip a restart-lost terminal for any turn other than the continuation it started > - The benefit is that back-to-back restarts no longer turn a recoverable lost turn into a failed run that needs Retry ## Linked Issues or Issue Description Refs #20107 **What happened?** Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade between builds plus repeated restarts) at |
||
|
DottaandClaude Opus 5.5
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
DottaandClaude Opus 5.5
|
15b987d9b8 |
fix(runner): a finish or block waits for held calls before it is judged (#20108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Restarts must be invisible to a task: an agent that was working
keeps working after the control plane comes back
> - Restart chaos t05 (Daytona warm wave, codex, K4 hold) ended a task
as blocked even though its document save succeeded
> - #20069 refuses a finish or block that the agent decided before it
saw results that had already returned. Here the results had not returned
yet when the block was checked: after the reattach, the held calls and
the later block are handled at the same time
> - This pull request makes a finish or block wait, for a bounded time,
for earlier calls of its turn that have not returned, and then apply the
same check
> - The benefit is that a controller outage during a Paperclip tool call
no longer turns into a false "blocked" task
## Linked Issues or Issue Description
Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261108T192459-s9501`, task t05 (Daytona warm, codex, C1, K4 hold of
182 s, master
|
||
|
DottaandClaude Opus 5.5
|
a8bbdf237c |
fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry
## Linked Issues or Issue Description
Refs #20089
**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.
**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.
**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
a88e23fa34 |
fix(runner): read an uncommitted turn start as empty history when the event window is complete (#20107)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run must survive repeated control-plane restarts; each
restore reconciles the turn its checkpoint names as active
> - That reconciliation reads the turn's history. When the provider has
no paginated history, the runnerd transport rebuilds it from the
committed runner events
> - After back-to-back hard restarts, the checkpoint can name an
accepted turn whose start was never committed. The runner was killed
after journaling `turn.start` and before the provider emitted anything
> - The rebuild treated every missing start as lost history and failed
with `codex_history_incomplete`, so the task needed a Retry
> - This pull request treats a missing start as empty history when the
committed window is provably complete
> - The benefit is that a run survives back-to-back restores, as it
already survives one
## Linked Issues or Issue Description
Refs #20074
**What happened?**
In restart-chaos wave w-20261108T200620-s9701 (local, K4/K6 + M1,
back-to-back K4 restores), three runs failed on the second restore. Two
were Claude ACPX C1 tasks; the third was an M1 parent with its subtask.
They failed with `native_session_recovery_failed:
HarnessReconciliationError: codex_history_read_failed: thread/items/list
requires supported paginated history; Error: codex_history_incomplete:
requested turn start is outside the retained runner event window`, and
each needed a Retry.
The sequence:
1. The first restore lost the in-flight turn and started a continuation
turn. ACPX echoes the requested turn id, so the checkpoint named it as
active.
2. Six seconds later a second K4 SIGKILL killed the runner. The runner
had journaled that `turn.start` (now `execution_indeterminate`), but the
provider had emitted nothing, so no `turn.started` was committed and the
runner outbox was empty.
3. On the second restore, `thread/items/list` for that turn hit the
guard, although the committed window was complete: source sequences
1–25, nothing evicted.
**Expected behavior**
The second restore treats the never-started turn like the first restore
treated its lost turn. The runner reports it lost on restore, and
recovery starts a continuation with no Retry.
**Steps to reproduce**
1. Have a checkpoint name an accepted turn whose `turn.started` was
never committed. For example, kill the runner after `turn.start` and
before the provider emits anything.
2. Restore the session. Before this change, recovery fails with
`codex_history_incomplete`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
3bf51913ef |
fix(daytona): gzip transport compression for sandbox images without zstd (#20104)
## Thinking Path - Per-turn remote sandboxes are the default, so every remote turn uploads the workspace into a fresh Daytona sandbox. - The Daytona plugin compresses large uploads with zstd, but only when the sandbox has a `zstd` binary. Without one, the upload goes raw: 220 MB for the paperclip seed. - The default Paperclip runner image has zstd, so it is already covered. A sandbox image without zstd is not: for example a user-supplied `config.snapshot` built on plain `debian:bookworm`. Such an image still has gzip. - This adds gzip as a fallback codec for those images. It is a robustness fallback, not a speedup for the default image. ## Linked Issues or Issue Description Refs #20100 **What happened?** On a Daytona sandbox image without a `zstd` binary, every remote turn uploads the workspace seed uncompressed. Measured live with the paperclip seed: raw takes 18.6–28.8 s, against 6.1–16.0 s compressed. **Steps to reproduce** 1. Use a Daytona snapshot whose image has gzip but no zstd. 2. Run a remote task on a large repository and inspect the `transfer` span: there is no `transfer.compression.codec` attribute, and the full raw size is uploaded. **Expected behavior** Large workspace tarballs are compressed in transit whenever the sandbox can decompress them. ## What Changed - `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`: - The existing availability probe (the same round trip as `mkdir`) also reports `gzip`. - Codec choice: zstd when the sandbox has it and Node supports it (unchanged), otherwise gzip, otherwise raw (unchanged). - gzip runs at level 1. On the 220 MB paperclip seed it produces 69.6 MB in 1.26 s on the host; zstd -3 produces 57.6 MB in 0.54 s. - In the sandbox, gzip decompresses with `gzip -dc -- <scratch> > <raw scratch>`. It reuses the existing explicit `chmod`, atomic `mv -f`, failure sweep, private host temp directory, and minimum-saving fallback. - A failed or partial decompress exits 49, the raw scratch is never promoted, and every scratch file is swept. The workspace sentinel and `baselineSha256` checks still run before `turn.start`. - The `transfer.compression.codec` span attribute is now `zstd` or `gzip` (still a closed set). - Tests in `file-sync.test.ts` (red on master): - A real shell promotion on a sandbox that reports no zstd: the upload is `.gz` and under half the size, `gzip -dc` runs, and the target is byte-identical with mode 0600 and no scratch left. - A truncated gzip upload fails the sync, never creates the target, and leaves no scratch. - Codec preference: zstd when both are present, gzip when only gzip is, raw when neither is. ## Verification - `npx vitest run` in `packages/plugins/sandbox-providers/daytona`: all pass. The existing zstd suite runs on this host because it has zstd. - `tsc --noEmit -p packages/plugins/sandbox-providers/daytona`: clean. - Live Daytona run on the runner image, 2 vCPU, 220 MB seed, through the real `performSyncIn` with each codec forced through the probe. The sha256 matched every time. | Codec | Times | |---|---| | raw | 28.8 s, 18.6 s, 20.2 s, 22.8 s | | zstd | 6.9 s, 6.1 s, 16.0 s, 8.5 s | | gzip | 9.8 s, 16.8 s | Network variance is large. Both codecs roughly halve the raw upload or better. ## Risks - **Host CPU:** about 1.3 s of streamed gzip per large upload, on the libuv thread pool, and only when the sandbox lacks zstd. Files under 8 MB, or that save less than 10%, still go raw. - **No compressor at all:** a sandbox without gzip or zstd uploads raw, as today. - **Probe caching:** the probe runs on every sync, on the `mkdir` round trip it already makes, so it adds no latency. It stays uncached, following the existing design's rule that one principal's observation is never reused for another. - **Observability:** one new value, `gzip`, in an existing closed-set span attribute. The endpoint gate is unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
015d3d0a7c |
fix(environments): tear down retain-on-failure sandboxes when their environment is deleted (#20085)
## Thinking Path - Remote sandboxes cost money while they exist, so every sandbox Paperclip keeps must have an owner and an end. - Two lease policies keep a sandbox after its run. `reuse_by_environment` keeps a warm sandbox for the next turn. `retain_on_failure` keeps a failed run's sandbox, stopped, as a workspace-restore source. - Environment deletion is the last chance to end both: with the operator's consent it destroys the environment's kept sandboxes, otherwise it refuses. - It only looked at `reuse_by_environment`. A `retain_on_failure` sandbox neither blocked the delete nor got destroyed, so it was orphaned the moment its environment was deleted. ## Linked Issues or Issue Description Refs #20048 **What happened?** In Daytona warm wave w-20261108T170423-s9501, the harness deleted the environment after grading. Three stopped sandboxes (4e20c8b3, abf0980b, 5e49c561) remained afterwards and had to be deleted by hand through the Daytona API. All three belonged to `retain_on_failure` leases: failed runs' sandboxes kept as workspace-restore sources. **This leaks on master today, with per-turn sandboxes too.** The four leases in that wave carried `workspaceRestoreRecovery` from `legacy-workspace-restore-recovery.ts` (no `nativeSourceRetention` marker). That writer turns any failed run's lease with a workspace-restore failure into a `retain_on_failure` restore source, whether the environment is warm or per-turn. Every such sandbox has been surviving environment deletion and costing money until deleted by hand. **Steps to reproduce** 1. A run in a sandbox environment fails, and its lease is released as a `retain_on_failure` restore source (sandbox stopped, not destroyed). 2. Delete the environment with `destroyReusableSandboxLeases=true`. 3. The delete succeeds; the restore-source sandbox is never destroyed and has no lease or environment left to find it by. **Expected behavior** The delete blast radius counts kept restore-source sandboxes like reusable ones. Without consent, the delete is refused; with consent, they are destroyed through the same claimed teardown before the environment row is removed. ## What Changed - `server/src/services/environments.ts`: `getDeleteBlastRadius` counts leases with policy `reuse_by_environment` **or `retain_on_failure`** (status `active`, `released`, `retained` or `failed`) as kept-sandbox holders. The blocked reason is still `reusable_sandbox_lease`. - `server/src/services/environment-runtime.ts`: `destroyReusableSandboxLeasesForEnvironment` selects both policies. It claims through `destroyClaimedSandboxLease` with `anyLeasePolicy` plus a scope condition limited to those two policies, keeping its existing guards: live-run skip, pending export, legacy shutdown protection, and the claim compare-and-swap. - `server/src/__tests__/environment-runtime.test.ts`: a released `retain_on_failure` lease on a failed run now blocks the delete and is destroyed by the consented teardown (`expired`, `environment_deleted`). This fails on master. ## Verification - `cd server && npx vitest run src/__tests__/environment-runtime.test.ts src/__tests__/environment-service.test.ts src/__tests__/environment-routes.test.ts`: 347/347 pass. - The new test fails on master: the blast radius reports no kept sandbox. ## Risks - **A delete now refused that used to succeed.** An environment holding restore-source sandboxes is refused until the operator consents to destroying them. That is the intended behaviour; it was silently leaking them before. - **Losing restore files.** The consented delete destroys restore sources, which is what the operator agreed to by deleting the environment. Leases with a pending export or legacy shutdown protection are still skipped and keep blocking. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a9ae1b69c3 |
fix(runner): let a finish through after a restart lost the provider turn (#20088)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner relays an agent's Paperclip tool calls and keeps them across control-plane restarts > - #20069 refuses a `paperclip_finish` or `paperclip_block` decided before an earlier returned result reached the agent > - The guard filters by turn id, but each committed event carries the run's one identity turn id, so provider turns are not separated > - When a restart loses a provider turn, a call of that turn gets its result but never its provider completion > - The guard then refuses every later finish of the run, so the run exhausts its retries and the user must click Retry > - This pull request makes a new provider turn end the pending calls of the previous turn > - The benefit is that an agent can finish after a restart that lost its turn ## Linked Issues or Issue Description Refs #20069 **What happened?** A restart chaos storm (local, Codex, A1/C1, K2 drains for 30 minutes, 116 tasks) had one failure. Task t110 needed a user Retry. Its run was drained twice. The first drain lost the provider turn while a `read_agent_instructions` call was pending. After that, all four `paperclip_finish` calls failed with `terminal_call_before_results` and named `read_agent_instructions`. The run ended with `native_session_retry_exhausted`. **Steps to reproduce** 1. Start a Codex run that calls a Paperclip tool. 2. Drain the control plane while that call is pending, so the Runner restores the session and reports `provider_turn_lost_on_restore`. 3. Let the agent continue in the new turn and call `paperclip_finish`. **Expected behavior** The finish is recorded. The lost call belongs to a turn that the agent will not see again. **Actual behavior** The Runner refuses every finish with `terminal_call_before_results`. The run fails and needs a user Retry. ## What Changed - `unseenReturnedToolCalls` forgets the calls it tracked when a `turn.started` event begins a new provider turn. - A new test reproduces the t110 event order: a call in a lost turn, then a new turn that reads, sees its result, and finishes. ## Verification - `npx vitest run src/live/terminal-call-freshness.test.ts` in `packages/paperclip-runner`: the new test fails before the change and passes after it (7/7). - `npx vitest run src/live/runnerd-codex-transport.test.ts -t terminal`: 10 passed. - `tsc --noEmit -p packages/paperclip-runner`: clean. - Evidence: the kept Runner control-plane state of t110 shows the call `exec-0f3d1b90` (generation 1) with a `semantic_tool.result` command but no `tool.execution.completed`, followed by `turn.failed provider_turn_lost_on_restore` and new `turn.started` events. ## Risks - The t03 case of #20069 stays covered: the stale block and the earlier calls are in the same provider turn, with no `turn.started` between them. The existing t03 tests still pass. - If a driver emits `turn.started` inside one provider turn, the guard can miss a stale call of that turn. No current driver does that. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4f7eee10c2 |
feat(recovery): let the board confirm an unrecorded owner stopped for every settling process kind (#20084)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Recovery releases in-progress records (leases, claims, intents) only
when it can prove their owner is gone
> - A record saved before its server process was fully recorded is
`unprovable`, and it is never released on its own
> - #20067 added a board-only action to confirm that such an owner
stopped, but only the environment run lease used it
> - Other settling kinds (for example the legacy controller lease, which
shows `missing_identity` rows in the Phase 2 reconciler data) stay stuck
with no way forward
> - This pull request opens the same confirmation to every settling kind
whose record names a partly recorded server process
> - The benefit is that an operator can release these stuck records
safely, and recovery settles them at once
## Linked Issues or Issue Description
Refs #20067, #20054
**What happened?**
The Phase 2 reconciler data shows `legacy_controller_lease` records that
stay `unprovable` with `missing_identity`: they were saved before
Paperclip stored the controller's PID space. Recovery never releases an
unprovable owner, and only the environment run lease offered the board's
"confirm the previous server stopped" action. These records, and records
of other settling kinds saved the same way, had no path forward.
Also, a board confirmation did not wake the reconciler. Unprovable
records are not watched, so the release waited for some unrelated pass.
**Expected behavior**
Recovery asks the board on the record's task. After the board confirms,
recovery releases the record through the kind's own compare-and-swap
settle, at once. A confirmation never releases a record that changed
since the board was asked.
**Steps to reproduce**
1. Have a legacy run whose `controller_process` was recorded without a
host hash (an older server).
2. Run the ownership reconciliation pass. The record is `unprovable`
(`missing_identity`) and no board action appears.
**Paperclip version or commit**
master at
|