2343 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	09241442ea	fix(server): let a native reviewer approve its review stage (#20373) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution policies can require an independent review stage before a task is done > - A native (Runner) agent that is the active reviewer had no way to record its decision > - `PATCH status: "done"` is refused for a native run that owns the task (#18411), and `paperclip_finish` accepts the report but leaves the stage pending > - The reviewer then searches the API until a deadline, and the task stays in review with nobody able to decide it > - This pull request records the final-stage approval on `PATCH` without releasing the run's lock, and lets the run's own finish report set the final status > - The benefit is that an agent reviewer can actually approve its review stage, in one short turn ## Linked Issues or Issue Description Refs #18411 (added the native status guard). Related: #20264 (explicit stage-decision operation, still open; this change is narrower and compatible with it). **What happened?** A native reviewer agent was woken as the active `review` participant. It read the deliverable and wrote its review in about 100 seconds. It then called `paperclip_finish` with `done`. The tool answered "Completion report accepted; the task's configured review stage is still pending". It then tried `PATCH /api/issues/{id}` with `status: "done"` and a comment. The route refused it with `native_run_status_reported_by_finish`. The agent searched the API for 2 more minutes and found no other path. The stage stayed pending. When the run ends with `done`, the status arbiter sees the pending stage and keeps the task in review (`governed_gate_pending`), so the stage can never resolve. **Expected behavior** The active review/approval participant can approve its stage from a native run. The task reaches `done` after the approval. The implementer still cannot approve. A run that is not the active participant still cannot change the task status. **Steps to reproduce** 1. Create a task with an execution policy that has one `review` stage with a native agent participant. 2. Move the task to `in_review` so the stage is pending and the reviewer agent wakes. 3. In the reviewer run, call `PATCH /api/issues/{id}` with `{ "status": "done", "comment": "Approved: ..." }`. 4. Observe `409 native_run_status_reported_by_finish`. Call `paperclip_finish` with `done` and observe that the stage stays pending. ## What Changed - `PATCH /api/issues/:id`: when the requester is the native run that holds the task lock and is the active participant, a `status: "done"` request on the final stage now records the approval decision (`executionState` completed, decision row, activity). The route removes `status` from the write, so the task stays `in_review` and the run keeps its execution lock. The run's `paperclip_finish` report then sets `done` through the normal status arbiter. - An intermediate-stage approval (which would hand the task to the next participant) and every other status change from such a run keep the existing `native_run_status_reported_by_finish` refusal. - `paperclip_finish` from the active participant (`done` or `needs_review`) while its stage is still pending is now rejected with the exact decision path, instead of being accepted with "stage still pending". - The native finalizer now treats the final-stage approval that the run itself recorded (read from the `issue_execution_decisions` row, bound to this run and agent) as the completion authority for its clean `done` report. Before this, a reviewer that cited free-text evidence had its report graded as incomplete, and the task was reopened as `in_progress` with a new wake for the reviewer. Failed verification, blocking remaining work, attention requests, other dispositions, and a still-pending stage keep the old outcome. - The reviewer/approver wake text names the decision path (`PATCH` with `status` and a required decision comment) and says a completion report does not record the decision. A Runner reviewer that needs changes is told to report them with `paperclip_block`, because a changes request from the run that owns the task is a reassignment that stops that run. - The locked review-policy recheck treats the deferred approval as a verdict, so a policy changed to `human_only` before the locked write still refuses it. ## Verification - `pnpm exec vitest run server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts server/src/__tests__/issue-execution-policy-routes.test.ts server/src/__tests__/issue-comment-reopen-routes.test.ts server/src/__tests__/issue-execution-policy.test.ts packages/adapter-utils/src/server-utils.test.ts`: 5 files, 595 tests pass. - `pnpm exec vitest run server/src/services/native-runtime/native-completion-feedback.test.ts`: 29 tests pass (embedded PostgreSQL), including the decision-row binding for `executionStageApprovedByRun`. - `pnpm exec vitest run server/src/services/native-runtime/status-arbiter.test.ts server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts server/src/services/native-runtime/native-stale-owner-completion.test.ts`: 72 tests pass. - New route tests fail without the route change (409 instead of 200/422) and pass with it. The new feedback test fails without the feedback change ("Completion report accepted" instead of a rejection). - `npx tsc --noEmit -p server` and `-p packages/adapter-utils`: pass. - A route test changes the review policy to `human_only` between the first check and the locked write; the deferred approval is refused (403). - Live full-stack chat eval `task-coordination.independent-review` (Runner, OpenCode) with this change: passed in GitHub Actions run 42705964209 (eval branch with the first two commits cherry-picked). The reviewer run recorded its approval 24 s after its review brief was released, reported `done` 14 s later, and the finalizer set the task to `done` with reason `execution_stage_approved`; total reviewer run 73 s. The earlier run 42702224713 exposed the finalizer evidence issue that the second commit fixes. ## Risks - Low to medium. The change is limited to one request shape: a native run, holding the lock, that is the active participant of a pending final stage, sending `status: "done"`. All other requests keep the old guard. - Between the approval and the finish report, the task shows `in_review` with a completed `executionState`. If the run fails before it reports, the existing native recovery owns the task, as for any other run. - Previous-release story: no persisted shape changes. A reviewer run that started before the deploy could not have recorded a deferred approval (the old route refused it), so the new finalizer input is false for it and the old outcome applies. - Intermediate-stage approvals and changes requests from native runs are still refused by the route. A Runner reviewer reports required changes with `paperclip_block`, which keeps the stage pending for a person. A self-service changes request needs the deferred handoff semantics in #20264. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 17:11:42 -06:00
DottaandPaperclip	facd409073	perf(workspaces): index legacy workspace companion lookups (#20372) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration must retain prepared sources and recovery holds. > - Companion discovery reads saved workspace references from run profiles. > - The selector can scan large saved profiles even when it returns few rows. > - This pull request adds one company and workspace expression index. > - Companion discovery uses the same expression as existing migration lookups. > - The result preserves each source check and reduces history scan work. ## Linked Issues or Issue Description **What happened?** The companion selector used a JSON path expression with no matching index. A read-only plan on about 22,000 run rows showed a sequential scan over a table with about 1.3 GB of table and TOAST storage. A repeated inspection reached its statement timeout. Existing migration lookups use a different JSON operator form, so one expression index must match both callers. **Expected behavior** Use a company-scoped indexed lookup. Keep prepared and finalized references, the exact text comparison, native issue references, and run order. Keep all source and recovery validation. **Steps to reproduce** 1. Retain many run profiles with saved workspace references and wide JSON fields. 2. Inspect a workspace with few matching companion runs. 3. Explain the company and saved-workspace selector. Without the new index, it can scan run history and read unrelated JSON values. **Paperclip version or commit** Baseline: `0e3fdebea8257bdff11573081c76b306f12db230`. **Deployment mode** Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1. The observed server plan used PostgreSQL 17.11. Related: [batched companion discovery in #19735](https://github.com/paperclipai/paperclip/pull/19735). The existing batching remains in place. The duplicate search found no open workspace index change. This branch also carries the isolated API route-summary correction shared with [#20370](https://github.com/paperclipai/paperclip/pull/20370). ## What Changed - Add one concurrent index on company ID and the saved native workspace ID expression. The index has no state, runtime, or status predicate. - Use nested JSON operators in companion discovery. Existing migration lookups already use this expression. Keep both accepted states and the same ordering. - Register the migration on the existing top-level migration path. Retain its advisory lock, interrupted index repair, and completion journal behavior. - Add a real PostgreSQL migration and ordered-result test. Update the existing companion query counter for the equivalent SQL form. - Generate the schema snapshot and retain the five newest snapshots. - Describe the task case-list route as “Find the cases for a task.” This keeps the route on the first discovery search page. Search logic and test assertions are unchanged. ## Verification - `pnpm --filter @paperclipai/db exec vitest run src/heartbeat-workspace-companion-index.test.ts --maxWorkers=1`: **1 passed**. It builds the concurrent index through the canonical migrator, checks index validity, preserves prepared/finalized and cross-issue/null-issue results, excludes foreign and malformed references, and checks one journal entry after a second migration call. - `pnpm --filter @paperclipai/server exec vitest run src/services/remote-workspace-online-source.test.ts -t 'discovers all Git companions' --maxWorkers=1`: **1 passed; 86 not selected**. This uses the real source owner and retains finalized recovery holds, company isolation, and a fresh obligation between inspections. - `pnpm --filter @paperclipai/db typecheck`, migration numbering/safety checks, and `git diff --check`: **passed**. - One small local SQL comparison used the captured baseline selector, the changed selector, 3,012 synthetic rows, about 37.8 MB of table storage, and the new expression index. Ordered results were identical. One client timing changed from **8.70 ms to 0.86 ms**. This is one local warm-cache comparison, not a server latency claim. Hosted improvement is **unverified**. - Initial setup attempts failed before the affected test ran: registry access, local tsx IPC, and a missing SDK build. The canonical Node 24.22.2/pnpm 9.15.4 setup and SDK prerequisite resolved these setup failures. The local SQL timing client used Node 26.4.0; the focused application tests used Node 24.22.2. - [The first CI run](https://github.com/paperclipai/paperclip/actions/runs/42705129724/job/129460702073) failed the inherited case-history API first-page search assertion. The same failure was reproduced on the unchanged base: the route ranked sixth for “case for task.” The isolated route-summary correction from #20370 passed the same three assertions. The search algorithm and assertions were not changed. - No public command changes require a documentation update. The schema and migration record the added index. No hosted DDL or deployment was performed for this PR. - Final head `c50101dabf46ade42b545d2051d724d93f413ccd`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42707623920) passed with 60 successful checks and three configured skips. [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/20372#issuecomment-6752305627) is 5/5 on this head with no open threads. ## Risks - The index adds storage and write cost when a run profile changes. - A concurrent build reads the table and can wait for old transactions. The existing migration path runs it outside a transaction and can repair an interrupted invalid index. - Existing nested-operator lookups can use the index. The companion selector needs this code change to use it. Other JSON path queries are unchanged. - The small local result does not establish end-to-end import speed or PostgreSQL 17 hosted latency. Fresh deployed measurement remains necessary. ## Model Used OpenAI Codex, GPT-6 family. The exact runtime model ID and context window are not exposed in this session. Used reasoning, code editing, shell tools, and local PostgreSQL tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 16:17:13 -06:00
DottaandClaude Opus 5.5	1baac27888	feat(server): chat bootstrap reads for instant task opens (#20355) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from the task list to see the latest messages and to reply to an agent > - Before the task chat shows its first frame, it waits for about ten separate reads: the issue, recent comments, comment input facts, recent activity, recent runs, interactions, attachments, live runs and the active run > - On a busy instance each read waits for the database, and the slowest read decides when the chat appears. A cold open takes 1.5–4 s on staging > - This pull request adds one read that returns all of that data for up to ten tasks, in the same JSON shapes as the existing endpoints, plus a single-task form > - The benefit is that the task list can preload its top tasks, and a cold open needs one round trip instead of about ten. The UI change is in the stacked pull request ## Linked Issues or Issue Description **Problem or motivation** A task open waits for about ten reads. With the database under load, these reads queue, and a cold open takes seconds. The list cannot preload a task cheaply, because each task costs about ten requests. **Proposed solution** `GET /api/companies/:companyId/issues/chat-bootstrap?ids=` returns the first-frame data for up to 10 tasks. `GET /api/issues/:ref/chat-bootstrap` returns it for one task. Each field equals the matching single endpoint's JSON exactly. **Alternatives considered** - Prefetch the existing endpoints for each list row. For 30 rows this sends about 210 requests to a database that is already the bottleneck. - A new projection with different shapes. This would duplicate client merge logic, and freshness rules could drift. **Roadmap alignment** This is part of the task-loading performance plan in #20276. It follows #20277, #20284 and #20286. ## What Changed - `packages/shared`: - `ISSUE_CHAT_BOOTSTRAP_MAX_IDS` (10). - `issueChatBootstrapQuerySchema`: UUIDs or identifiers, normalized, deduplicated and strict. - The `IssueChatBootstrap` and `IssueChatBootstrapItem` types. - New `server/src/services/issue-chat-bootstrap.ts`: - Resolves the refs inside the company and applies the `issue:read` decision to each task. - Reads the tasks with a concurrency of 2, and returns them in request order. - `server/src/routes/issues.ts`: - The core issue read and the recent-comments read are now shared helpers, so the bootstrap and the single endpoints use the same code. The core view keeps its Server-Timing header. - Adds the batch route and the single-task route. - Excludes the single-task route from the issue ETag middleware, so a fetch without validators always gets a 200 with `Cache-Control: private, no-store`. - New `server/src/services/issue-run-presence.ts`: - The live-runs and active-run payload builders, moved from `agents.ts` without change. - The existing routes call them, and their behavior is unchanged. - `server/src/routes/openapi.ts`: entries for both routes. ## Verification - `cd server && npx vitest run src/__tests__/issue-chat-bootstrap-routes.test.ts` passes 17 tests. They run on embedded Postgres with privacy enforcement on. They cover: - Each field deep-equals its single endpoint's JSON, for an ordinary task and for a conversation task. - The single-task route equals the batch item, by UUID and by identifier. It is never answered with a 304. - Items keep the request order, with UUIDs and identifiers mixed. - Private, other-company, missing and unknown tasks are omitted, and none of their data appears. The single-task route returns 404 for them. - Malformed, extra or repeated parameters return 400. More than 10 ids also return 400. - A task-bridge key gets 403. - A 10-task request writes nothing: no inserts, updates or deletes, and no activity-log or comment changes. - The issue, agent, activity, live-run, cold-open, summaries, ETag and OpenAPI route suites pass: 56 files, 1,771 tests. - `cd server && pnpm typecheck` passes. - Query cost, local embedded Postgres, one 10-task request: about 49 SQL statements per task, 55–130 ms in total. A test fails if it goes above 60 per task. ## Risks - **Database load:** the stacked UI preloads up to 30 tasks, about 1,500 statements at idle time. This is the same work as opening those tasks, but it is done before the person clicks. The work runs in sequential requests of 10, with 2 tasks at a time. Most of the cost is existing run decoration. Live runs and the active run decorate the same running run, so that work runs twice. Sharing it is a follow-up. - **Stricter access:** live runs and the active run now apply the `issue:read` policy inside this endpoint. The existing routes keep their company-only check. - Hidden tasks are returned, as `GET /issues/:id` returns them. - No migration and no persisted-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing, subagents). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 15:05:41 -06:00
DottaandPaperclip	c67909cb0f	fix: keep tool approvals valid through OAuth refresh and delayed review (#20340) ... ## Thinking Path > - Paperclip lets people manage agent work and approve app actions. > - An approval must bind the action that the person reviewed. > - OAuth access tokens can expire while that person is away. > - The old target snapshot treated each token refresh as a new authorization. It also imposed a one-hour review deadline. > - This pull request separates token renewal from authorization identity and keeps the human decision durable. > - Current access, the reviewed arguments, meaningful target changes, task closure, and execution receipts still control dispatch. > - People can approve later without repeating a decision after routine connection maintenance. ## Linked Issues or Issue Description **What happened?** A pending app action returned `approved_tool_target_changed` after its OAuth tokens refreshed. The tool and connection settings were unchanged. The same card also expired after one hour. A sign-in problem could discard an accepted decision before provider dispatch. **Expected behavior** The original card remains reviewable after routine OAuth refresh and a server restart. A later approval runs the exact reviewed action once. A connection repair preserves the decision while Paperclip verifies the same authorization. Meaningful changes require a new review. **Steps to reproduce** 1. Connect an OAuth MCP app and configure an action to require approval. 2. Request the action, then refresh its access and refresh tokens before approving. 3. Approve the original card. Before this change, it returns HTTP 409 even when the authorization is unchanged. 4. Wait more than one hour. Before this change, the card expires without a task-specific deadline. **Paperclip version or commit** Reproduced from master at `76254ead2229`. The final branch is rebased onto current master. ## What Changed - Add versioned approval target comparison. New snapshots bind credential identity epochs, consent, permissions, ownership, connection configuration, tool contracts, and credential references. Authenticated OAuth refresh preserves identity. Manual credential replacement does not. - Remove the implicit one-hour deadline from ordinary app action reviews. Keep explicit deadlines and terminal-task expiry. Synchronize the nullable interaction deadline in shared contracts and UI. - Keep accepted, undispatched actions when sign-in is required. Show a saved-approval repair state. Resume on committed connection changes after target and access revalidation, without an idle retry loop. - Preserve the existing execution claim, result receipt, and uncertain-outcome no-replay rules. - Explain meaningful target changes without inviting the person to retry the stale card. - Add a production task-thread Storybook example and document compatibility and recovery behavior. ## Verification - Red: production-refresher invalid-grant handling and formal-approval withdrawal during a sign-in wait also reproduced as failing regressions before their fixes. - Red: before implementation, the local OAuth HTTP fixture failed for both public and confidential clients with `approved_tool_target_changed` after token refresh. - Green: the extended fixture passes for public and confidential clients with personal and shared grants. It covers actual local HTTP token exchanges, multiple refreshes, a rebuilt gateway, an ended originating run, approval after 25 hours, duplicate approval, preserved sharing arguments, sign-in waiting, and event-driven repair. The gateways use the production OAuth refresher; an invalid refresh preserves the accepted decision before provider dispatch. Omitted-scope responses cover equivalent scope-reporting changes. The provider is an isolated local fixture, not a deployed Notebook service. - Negative controls refuse changed consent, scopes, tool definition, destination, and manual credential replacement before provider execution. - Existing gateway service, connected-tool acceptance, secret owner, secret service, unsigned-review queue, and connection-health suites passed. Explicit-deadline fixtures still prove expiry behavior, including deadlines on accepted sign-in waits. A withdrawn formal approval cancels a waiting action before connection recovery. - Approval card, task-chat, error-copy and target-comparison tests passed. Server and UI typechecks passed. Token gates and Storybook catalog checks passed. - Inspected the new saved-approval repair card in local Storybook using the production task thread. It shows a reconnect link and no second approval button or false running state. - `pnpm verify --explain` returned exit 2 and requested broad CI coverage. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507) passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build, typecheck/release registry, general and serialized tests, and all eight browser shards. - Greptile reviewed the same final head at 5/5. Both findings from the first review are fixed and their threads resolved; no open follow-ups remain. ## Risks - No database migration is needed. The new comparison reuses the vault identity epoch. Only authenticated OAuth refresh code can preserve that epoch; HTTP credential-write schemas cannot select that behavior. - Earlier signed requests keep their original exact-version checks and saved deadlines. Already-expired cards are not revived. A historical card whose credentials already changed needs a fresh review. - A new consent, account, permission set, destination, or manually replaced credential must not inherit the old authorization. Reconnection continues automatically only when revalidation proves the same saved authorization. - The interaction enrichment now permits a null deadline. Existing string deadlines remain valid. The web client and shared validator change together. - This PR does not deploy to staging or perform a live Notebook publication. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 13:46:28 -06:00
DottaandPaperclip	0e325e8d52	fix(runner): settle saved handoffs after verified shutdown (#20336) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 13:46:16 -06:00
Nicky LeachandPaperclip	79bf733012	fix(runner): share the bounded ACPX cleanup proof (#20332) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner keeps a lifetime fence until the original ACPX provider stops. > - A recovered Runner must prove that fence is free before it clears saved cleanup state. > - Suspension and other recovery paths already wait for transient listener holders. Recovered shutdown used one immediate probe. > - That difference made parallel Rust tests fail after they dropped their original listeners. > - This pull request uses the existing bounded proof for recovered shutdown and removes the repeated test retry. > - The benefit is one cleanup rule, with the same fail-closed proof and no new work on normal task dispatch. ## Linked Issues or Issue Description Fixes: #20210 Thanks to @genius-0963 for the test-only fix proposed in #20321. That related alternative adds a test-only shutdown retry. This change applies the existing bounded cleanup proof in the shared shutdown path and removes the same retry pattern from native-provider integration tests. It also tests a single shutdown call while the original quorum is still held. ## What Changed - Use `await_provider_lifetime_fence` for a recovered executor with no live session and unconfirmed provider exit. Delete the unused one-shot helper. - Keep the acquired quorum held through the existing durable state write. A failed proof leaves `providerExitUnconfirmed` set. - Give the shared timeout message wording that applies to shutdown and suspension. - Extend the recovery test with a worker-start signal. Hold the original listeners until the single shutdown call is waiting, then release them and assert successful cleanup without a provider session. - Use the same bounded proof in the retired-quorum fixture test. Remove the duplicate retry loop from the native-provider test helper. - Document the bounded recovery cleanup wait. ## Verification - macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only targeted checks ran locally. - `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`: passed. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on the rebased head (5.21 seconds). The initial head also passed five more parallel-test runs (45 tests per run), with no failure. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test native_provider_backend acpx_`: 14 passed on the rebased head (9.43 seconds). These tests use the real Rust executor and controlled sidecars. - Negative control: restore only shutdown's original one-shot proof. The new regression fails at `shutdown must wait while the original lifetime still holds a quorum`. Restore the fix and the affected module passes. - Independent parent verification on current commit `368fc814f7`: the three exact recovery, held-quorum, and retired-quorum tests passed (3 tests, 5.06 seconds). Independent source review confirmed the unchanged three-file scope and found no actionable issue. - The active-turn recovery test still holds a live quorum through the five-second bound. It checks that shutdown refuses cleanup, saved state remains unconfirmed, and the provider never starts. It then releases the quorum and confirms cleanup. - Standard Greptile scored current commit `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads. - Initial GitHub general-server shard 7 failed in unchanged `remote-workspace-nested-verification.test.ts`, at its 15-second wait for a child-process import checkpoint (expected one row, got zero; both Vitest attempts). [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815). That fixture runs Node, PostgreSQL, Git, and storage migration; it does not run the native Runner or the changed Rust cleanup. The cause of the checkpoint timing miss remains unknown. The focused retry passed on the unchanged source ([successful retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)). This proves recovery, not the cause of the original timing miss. GitHub kept the reusable-workflow aggregate failed after that job-only retry and rejected both a final-gate retry and a failed-job retry. A subsequent run-level retry materialized only the gate and aggregate jobs; its aggregate inputs reported prior jobs as `abandoned`, so it failed without a new test failure. A new `full-ci` label event requests a fresh workflow graph on the same commit. The reason for those rerun-state results is not established. - The fresh `full-ci` run [42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208) then lost several runners mid-job. The maintained [runner-loss recovery 42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525) classified all four failed jobs as runner loss: typecheck, E2E shards 2 and 8, and server shard 4. The evidence was a shutdown log or lost-communication annotation. It retried the failed jobs together; canceled sibling jobs were also requeued. This is runner-interruption evidence, separate from the earlier checkpoint assertion. - The maintained recovery attempt again failed the same partial-checkpoint assertion in unchanged `remote-workspace-nested-verification.test.ts`, twice ([job 129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)). Neither checkpoint-failure job logged runner shutdown or lost communication; their resource summaries reported no significant constraint. The child phase at the timeout remains unknown. No further blind retry or TypeScript change is included here. Related startup-handshake work exists in #20339; its controlled-delay evidence establishes a deadline-boundary defect but does not establish the phase of these original CI failures. - Rebased onto current master after upstream #20339 merged. That separate change fixes a demonstrated child-startup/checkpoint-deadline boundary. It does not prove which child phase caused the earlier zero-checkpoint assertions. This PR still changes only its original Rust and documentation files. Fresh GitHub CI on the new base will establish recovery, not retrospective cause. - Full local tests and a local release build were not run. Current-head [PR CI 42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411), [Runner Windows 42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803), and [Docker Runner 42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601) all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check list has 59 successes and 5 intentional skips. GitHub supplied broad typecheck, build, and optimized Rust coverage. This is new-base recovery; the cause of the original checkpoint assertion failures remains unproved. ## Risks - A genuinely unconfirmed original provider can now hold recovered shutdown for up to the existing five-second proof deadline. The proof returns immediately when two candidate ports are free. - The changed branch requires both no in-memory provider session and unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch keep their existing paths. Outer controller close deadlines remain authoritative. - State saved by the previous release carries forward unchanged. No schema, launch identity, profile digest, protocol, or retry-budget change. - No live provider session or hosted deployment was tested. The tests cover cleanup proof, refusal, durable state, and controlled provider execution. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 11:36:44 -08:00
DottaandClaude Opus 5.5	4a349ea447	fix(chat): tell agents that instruction saves keep no history (#20328) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents can read and edit other agents' instruction files with dedicated tools > - #14420 changed instruction saves: Paperclip now keeps only the current file and records no revision history > - The agent-facing descriptions still describe a "canonical instruction revision" history > - In a live eval, a chat agent asked to undo an instruction change found an empty history, read it as a test artifact, guessed the earlier text from the current file's wording, wrote it, and reported a restore > - This pull request tells agents the real contract in chat guidance and in the tool and API descriptions > - The benefit is that an undo with no saved version leaves the instructions alone and asks the person, instead of inventing an earlier version ## Linked Issues or Issue Description Refs #14420 Refs #19515 (open: would list save records from the activity log. Those records hold no content, so they still cannot restore earlier text) **What happened** A person asked a chat agent: "Undo Morgan's last instruction change." Morgan's history was empty, because saves do not record revisions. The agent's reasoning said the history "is only populated with canonical revisions" and that there was "no revision tracking in this e2e harness". The agent then removed the last line of AGENTS.md and replied "Undone. Morgan's AGENTS.md is back to …". No saved source had the earlier text, so the agent guessed it. **Expected behavior** When there is no saved earlier version, the agent leaves the instructions unchanged. It says plainly that no earlier version was saved, and it asks the person for the text they want or which part to remove. **Steps to reproduce** 1. Create an agent with an AGENTS.md file. Edit the file once through `PUT /api/agents/{id}/instructions-bundle/file`. 2. `GET /api/agents/{id}/instructions-bundle/history?path=AGENTS.md` returns `{"revisions":[]}`. 3. Ask a chat agent to undo the agent's last instruction change. The agent rewrites the file from a guess. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says three things: - Saves keep only the current instruction files. - An empty history means no earlier text is retained. - An undo with no listed revision must leave the file unchanged, say that no earlier version was saved, and ask for the wanted text. - The `get_agent_instruction_history` contract now says that it lists only pre-upgrade revisions, and that an empty list means Paperclip retains no earlier version. - The `restore_agent_instructions` contract now says that only listed pre-upgrade revisions can be restored. - The OpenAPI summary for `GET /api/agents/{id}/instructions-bundle/history` says the same. - The semantic tool contracts, the protocol manifest and the seeded eval catalog hash are regenerated. - Regression tests check the new guidance and both live tool contracts. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts server/src/__tests__/board-access-denial.test.ts server/src/__tests__/codex-local-execute.test.ts server/src/__tests__/runner-api.test.ts` passes. - In `packages/paperclip-runner`, `pnpm exec vitest run src/semantic-tools/paperclip-semantic-tools.test.ts src/catalog src/evals/native-execution.test.ts` passes. - `node scripts/generate-semantic-contracts.mjs --check`, `generate-protocol-manifest.mjs --check` and `generate-replay-goldens.mjs --check` pass. - `tsc --noEmit` passes for `server` and `packages/paperclip-runner`. - The live result is the chat eval `agents.restore-instructions`, run with this change cherry-picked: run 42666621831 passed. The agent replied that Morgan's history is empty and no earlier version exists. It left AGENTS.md unchanged and asked for the exact text or the line to remove. Run 42660236086, without this change, failed: the agent guessed and rewrote the file. ## Risks - Low risk. Only agent-facing text changes, and no behavior changes. - The runner semantic catalog hash changes. This is the same kind of change as #19983. - If #19515 lands, history lists save records without content. The guidance wording must then say that listed saves without content cannot be restored. Whichever PR merges second must reconcile the two texts. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 12:23:12 -06:00
DottaandPaperclip	76254ead22	fix(runner): allow Codex interruption to settle governed waits (#20330) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - A native run must stop provider work before it waits for a human response. > - The Codex transport allows 30 seconds for a control command. > - The governed-wait loop used its 15-second default for Codex. > - A valid interruption terminal at 18 seconds therefore failed the run. > - This change gives Codex the existing 30-second terminal drain allowance. > - The run still fails if it cannot prove that the provider turn ended. ## Linked Issues or Issue Description **What happened?** A Codex run can save a human wait, accept an interruption, and fail before the provider sends its terminal. Its saved wait remains intact, but normal finalization does not occur. **Expected behavior:** Allow terminal delivery through the transport's bounded command window. Keep missing-terminal cases closed. **Steps to reproduce:** Run the new protocol regression against the old backend. The real Runner binary uses a controlled Codex process that delays its interruption terminal for 18 seconds. The run fails at the 15-second drain deadline. The same process passes with this change. **Paperclip version or commit:** Reproduced against `1b7fe33432`. **Deployment mode:** Local test of the native Runner path. The incident was observed in a hosted sandbox. **Installation method:** Source checkout. **Operating system:** macOS arm64 for local verification. Hosted incident runtime was Linux. **Additional context:** No matching repair was found. This addresses a reproducible deadline mismatch. The hosted log ends terminal observation at 15 seconds, so it cannot prove whether that provider would have settled within 30 seconds. The approved local live-provider check passed. This PR does not claim recovery of the hosted task. ## What Changed - Give Codex a 30-second governed-wait terminal drain through the existing backend option. - Keep other providers and the caller's shorter execution bound unchanged. - Add real Runner protocol cases for prompt, delayed, and absent terminal delivery. Check retained wait state and one provider turn. - Document the bounded wait and terminal-proof requirement. ## Verification - Red: the 18-second controlled-provider case failed with `governed_wait_provider_terminal_unproven` on the old backend. The prompt case passed. - Green: 282 tests passed across the native runtime, harness backend, and new Runner protocol suite, including a repeat on the rebased PR revision. `pnpm verify --explain` returned exit 2 because the execution-contract document requires broad CI; the [required CI run](https://github.com/paperclipai/paperclip/actions/runs/42668235142) passed on the final head. All 58 applicable checks passed; the three skipped checks were the two unrelated Storybook jobs and the selected-test planner replaced by broad coverage. - The missing-terminal case still fails without `completeRun` or a successful run terminal. The saved governed wait remains available. Only one provider turn and one tool call occur. - Existing runtime tests cover shorter deadlines, foreign or unproven terminals, complete usage without a terminal, and checkpoint recovery without another model turn. - Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64, disposable local workspace. One `request_human_input` tool call returned a pending interaction fixture. The real provider emitted `turn.interrupted`; the controller accepted the yielded wait and its run terminal. The check passed in 8.14 seconds. The tool callback and persistence port were controlled; no hosted server permission or Notebook publication claim is made. - Initial local probe setup failed before model execution: first the explicit launch lacked `app-server`, then it omitted the Runner permission configuration. The successful check used the Runner's normal launch arguments and isolated home with the approved existing account. - Hosted sandbox acceptance and recovery remain unverified. The 18-second delayed shutdown is a controlled protocol regression; the live model check proves normal shutdown, not that specific latency. ## Risks - A Codex provider that never stops can occupy the settlement window for up to 15 additional seconds. It still fails closed and is quarantined. - No persisted schema, provider identity, permission, or recovery rule changes. Previous-release checkpoints use the same format and existing recovery checks. - This change cannot fix a provider that never emits a terminal. It also does not grant missing setup permissions or decide pending approvals. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool calls, code edits, and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 12:11:47 -06:00
Dotta	a3e7e63330	Guard Native recovery with process-space proof and explicit containment (#19813) ... ## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. **The current fix has a causal red/green check.** On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: **5 passed, 0 failed;170 filtered rows excluded**. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed **25/25**, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, **15 selected accepted-result/restart controls passed**. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete **17-case maintenance-server file**. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed **29/29**. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 12:06:11 -06:00
DottaandClaude Opus 5.5	8fe524f796	fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a new attempt can start on a task, Paperclip must prove the previous run's agent stopped, so two agents never work the same task at once > - A local native run can be interrupted after it asked to start the runner but before it recorded the runner's process. The runner may have saved state and started a provider, and provider processes can outlive it, so the existing check (#19694) rightly refuses to assume the run stopped > - Today that case has no way out: every Retry is accepted and then waits forever as `process_identity_missing` (storage chaos ST1, t02/t03) > - This pull request makes the wait explicit and actionable. The task explains why it waits, and a board operator who has checked can confirm the run stopped; the parked Retry then starts at the click > - A second shape from the Daytona storage baseline (t06) parked a Retry with no reason at all: a follow-up was interrupted by a graceful shutdown while it was still restoring its workspace. No provider ran, but the restore hold kept the same agent held, so the Retry never started > - The benefit is that no Retry stays accepted-and-waiting with nothing anyone can do ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks t02 and t03. **What happened (CHA-3, run `3c3f6241`):** - The run requested a runner launch (`native.process_start_requested`), and a K2 restart hit before its process identity was recorded. - After the restart, the resumed attempt failed on the storage outage. The task was blocked. - The board's Retry (202) was parked as `executionWait.reason = process_identity_missing`: "The previous run has no verified stop record." - `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this run's runner saved control-plane state. Once that exists, a provider may have started (#19694), and a provider is not guaranteed to carry `--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on purpose). The shared owner prover needs a PID, which the run never recorded. - Nothing could ever clear the wait. **Daytona storage baseline `w-20261109T154810-s9911`, task t06:** - The run succeeded. The follow-up was cancelled (`continuation_task_ownership_changed`), and the next run was interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s into `preparing`. It was restoring the saved remote workspace and had launched no provider. - Restore recovery saved the files (`workspaceRecovery.status: recovered`, system recovery `complete`). But the `legacy_execution_requires_reconciliation` restore hold releases only for a different owner, or for a native Stop fenced before any provider turn, so it kept the same agent held. - The UI Retry returned 202 and parked with `executionWait.reason: execution_recovery`, the generic wait, so the task showed a Retry that never started. Automatic proof needs the runner to record its identity before saving state and to group provider processes under it. That is follow-up runner work; this PR removes the endless wait now. ## What Changed - `server/src/services/native-local-process-stop.ts`: new `confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished native run of the task with no recorded process and whose latest launch evidence is an unconfirmed launch on this host in a local environment. It still refuses while a runner with `--run-id <run>` is running. It records `native.local_process_stopped` with `basis: "operator_attested"` and the attesting user. That is the receipt the existing continuation gate already accepts. - `server/src/routes/issues.ts`: `POST /issues/:id/recovery-actions/confirm-run-stopped` (`runId`, `stopEvidence` of 20 to 12,000 characters). It is board-only and requires issue read plus `runtime:confirm_owner_stop`, like the other stop attestations. It logs `issue.run_stop_confirmed` with the evidence, promotes deferred intents for the issue, and resumes each parked task Retry, so work starts at the click. Refusals return 409 with a code. - `packages/shared`: `confirmRunStoppedSchema`, and the presentation user action `confirm_run_stopped`. - `server/src/services/task-interruption-presentation.ts`: a Retry parked as `process_identity_missing` for a native run that recorded no process now presents as waiting for the user (`canRetry: false`, `userAction: "confirm_run_stopped"`), with a message explaining why and who can act. A run with a recorded process keeps the existing verification wait. - UI: - `ExecutionBlockerNotice` shows the explanation and, for users who can confirm a stop, "Confirm it stopped" with a short evidence form. - It reuses `PreviousServerStopConfirmation`, now with optional `action`, explicit `runId` and `formId`, and a custom button label. - The Inbox names the action. - `server/src/services/execution-blocker.ts`: `interruptedPreparationBeforeProviderWorkCondition`. The restore hold also releases for the same agent when the stopped run was a system-interrupted preparation (legacy runtime never resolved, still `preparing`) with no process identity, no runner or provider events, no finalization or result, and every lease released with a confirmed stop. It mirrors the existing interrupted-preparation proof and still requires the completed workspace-recovery receipt. Auto-replay is unchanged; this only lets the user's Retry or message start. - Review fixes: - `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and lease checks, and the presentation offers `confirm_run_stopped` only when it passes. - The attested stop wakes cost accounting in the same transaction. - The confirmation form is keyed by task and run. - The route is registered in OpenAPI. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`: - The parked Retry presents `confirm_run_stopped` with the explanation. - Confirming records the operator-attested stop and the activity entry, satisfies the stop check, and resumes the parked Retry: its receipt is coalesced into a newly started run. - It refuses with 409 while a runner for the run is running. - Agents and viewers get 403. - `ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`: the notice offers "Confirm it stopped" (and no dead Retry) and posts the run and evidence. A viewer sees the explanation without the action. - `server/src/__tests__/interrupted-preparation-restore-retry.test.ts` (t06 shape): - The restore hold no longer blocks. - A Retry is accepted and resumed into a started run, and the audit hold keeps `replay: blocked`. - The hold stays when a provider turn started, when a runner event was recorded, when an operator stopped the run, or when the sandbox has no stop receipt. - `issue-confirm-run-stopped-routes.test.ts`: - A case per refusal code checks the presentation keeps the generic wait and the endpoint returns 409. - An `alreadyStopped` case. - An accounting-wake assertion. - The UI test switches targets with stale evidence. Regression and static checks: - `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control tests: 74 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A board operator can now attest that an unidentified run stopped. The attestation is permission-gated, requires written evidence, is activity-logged, and is refused while a runner for the run is visibly running. A wrong attestation could start a second agent beside an orphaned provider, the same trust model as the existing previous-server attestations. - The confirmation component now accepts a run without a recovery action. Existing callers pass the action as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 11:36:41 -06:00
DottaandPaperclip	ff2d463eed	feat(cli): manage agent grants and keep remote login on the public origin (#20323) ... ## Thinking Path > - Paperclip lets people control their AI agents and permissions. > - Remote operators need a browser-approved CLI login. > - Proxies can send that login's approval link to an internal host. > - Human member commands cannot grant an agent an explicit permission such as commentary:view. > - This change keeps approval on the canonical public origin and adds Board-only single-grant commands. > - The API checks company access and permission-management authority and records each change. > - An operator can sign in, grant one agent permission, and preserve the agent's other grants. ## Linked Issues or Issue Description **What happened?** A proxied CLI login returned an approval URL on the upstream host. The signed-in browser was on the public host. The CLI also had no command to add an arbitrary explicit agent grant: member commands manage human members, while agent permissions:update exposes fixed configuration fields. **Expected behavior** Approval uses the configured canonical public origin. A Board operator with users:manage_permissions can grant or revoke one explicit agent permission without replacing other grants or activating an inactive membership. **Steps to reproduce** Configure a public auth origin. Create a CLI challenge through a proxy that replaces forwarded-host headers. Inspect the approval URL. Then try to grant commentary:view to an agent through the CLI. ## What Changed - Use the existing public-origin resolver for CLI approval URLs. - Add agent permission:grant and permission:revoke commands. - Add PATCH /api/agents/{id}/permission-grants with strict shared validation and Board-only OpenAPI metadata. - Serialize each mutation on the agent row. Preserve unrelated grants and membership state. Commit the grant and its activity record together and publish activity after commit. - Document remote login and single-grant management. ## Verification - The new approval-link regression failed before the fix and passes after it. - All 25 CLI auth route tests pass. - All 72 affected CLI, login and OpenAPI tests pass. - All 5 PostgreSQL grant tests pass, including actual commentary reads (403 before grant, 200 after grant, 403 after revoke), preservation, audit attribution, concurrent additions, invalid inputs, inactive or missing membership, agent self-grants, cross-company denial, and rollback on audit failure. - Shared and CLI typechecks and server declaration-layer checks pass. - pnpm verify requests broad CI coverage because doc/CLI.md is a global verification input. Greptile reviewed final head 338c58fb24 at 5/5 with no findings. All 58 non-skipped checks pass on that head. CI run: https://github.com/paperclipai/paperclip/actions/runs/42663640985. Live target acceptance is in progress. ## Risks This adds an explicit permission-management API. It requires Board authentication, company access, and users:manage_permissions. Foreign-company agents return 404. Granting a key intentionally makes that key unscoped in its company; revocation removes that key. Other keys and agent configuration remain unchanged. Inactive memberships and unapproved or terminated agents cannot gain grants. No schema change is needed. Revert this change to remove the new management surface; existing grants retain their normal authorization rules. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, shell tools, and browser testing. The runtime does not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 11:35:52 -06:00
DottaandClaude Opus 5.5	97188f276c	perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner runs each agent's provider (for Claude, an ACPX sidecar) and keeps its session across runs. > - #20190 restarts the ACPX provider when a turn settles, so a warm follow-up does not pay the respawn on its critical path. > - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits with its run. Nothing reuses the generation that restart prepares: the next run starts a new Runner process and loads the session again. > - In that mode the restart can only spawn a provider for teardown. On a loaded host it can also fail, which was the cause of the closed sessions that #20285 fixed. > - This pull request skips the turn-end restart when the controller launched the Runner as per-turn. > - The benefit is that per-turn runs never spend a provider spawn they cannot use, and never risk a restart failure at the end of their run. ## Linked Issues or Issue Description Refs #20190, #20285 **What happened?** A per-turn Runner restarted its ACPX provider after its turn settled, although the Runner exits with its run. The next run launches a new Runner process (`runner.process.launch mode=resume`, then `provider.session.load`), so the restarted generation is torn down unused. In an earlier A/B on a host with load average about 17–27, this restart fired and failed, and the failure closed the session (fixed in #20285). **Steps to reproduce** 1. Run a Claude task with the default per-turn lifecycle. 2. Let the turn settle and the controller acknowledge its events before it shuts the Runner down (more likely on a loaded host). 3. The Runner reaps the provider, spawns a new generation, and then exits with the run, which reaps the new generation too. **Expected behavior** A per-turn Runner does not restart its provider after its turn. A warm Runner keeps the #20190 restart. A later turn of the same run still restarts the provider at its start. ## What Changed - `DurableRunnerConfig` has a new `turn_scoped` field. - `paperclip-runnerd` sets it from `--lifecycle-mode`, which the controller already passes (`durable-prp-control-plane.ts` pushes `--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only `per_turn` sets it. `warm`, and launches without the flag (older controllers), keep today's behavior. - `NativeProviderCommandExecutor::prepare_next_turn` returns early when `turn_scoped` is set. The lazy restart at the next turn start is unchanged. - Tests: - New `a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`: no new provider process and no sidecar command after the turn settles; a later turn of the same run still restarts at its start (`session.suspend`, `initialize`). It fails without the skip (2 process starts instead of 1). - New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing. - The other `DurableRunnerConfig` literals set `turn_scoped: false`. ## Verification - `cargo test -p paperclip-runner-core`: - `--lib`: 602 passed. - `--test acpx_park_preparation`: 22 passed. - `acpx_park_readiness`: 8 passed. - `acpx_warm_attachment`: 4 passed. - `acpx_provider_suspend`: 5 passed. - `acpx_provider_session`: 12 passed. - `acpx_provider_turns`: 29 passed. - `native_provider_backend`: 47 passed. - `durable_recovery`: 3 passed. - `--bin paperclip-runnerd turn_scoped`: 1 passed. - `cargo build --workspace --all-targets` passes. - Local live A/B with `ttft-baseline --adapters claude --samples 3 --restarts 0 --phases coldwarm`. Control is master after #20285 (7fe9345030); the other arm is this branch. Host load average was about 6–9. | | Cold first activity p50 | Cold TTFT p50 | Warm first activity p50 | Warm TTFT p50 | Result-accepted → exit | |---|---|---|---|---|---| | Control | 5.61 s | 8.86 s | 4.95 s | 5.00 s | 1.49–1.84 s | | This branch | 6.72 s | 9.83 s | 5.52 s | 5.57 s | 1.45–1.63 s (one cold run 3.64 s) | - At this load the difference is within run-to-run noise. Both arms passed 6/6 samples. - A process sampler saw no short-lived provider process in either arm. Every provider process lived 11–16 s, about one run. So at load ~8 the per-turn Runner exited before the turn-end restart ran, and this change had nothing to remove. - The restart did run, and fail, in the earlier A/B at load ~17–27 (#20285). This change removes that path for per-turn Runners. I did not reproduce that load level here, so this PR has no measured speed gain to claim. ## Risks - **Low risk.** The change only removes work in per-turn Runners. Warm Runners and launches without `--lifecycle-mode` behave as before. - If a per-turn run starts a second turn, that turn restarts its provider at its start, as before #20190. That costs the respawn on its critical path; per-turn runs normally have one turn. - Remote launchers that build their own `runnerd` arguments and do not pass `--lifecycle-mode` keep today's restart. That is the safe default. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:50:25 -06:00
DottaandClaude Opus 5.5	fd2e7482c0	fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A local native run uses one runner daemon (`paperclip-runnerd`). runnerd starts the provider (for example OpenCode), and the provider starts tool processes > - When runnerd dies in the middle of a tool call, the provider and its tools are re-parented. The only record of that process tree was in runnerd's memory > - Close then cannot prove that those processes stopped. It quarantines the session, and the next attempt stops with `native_session_cleanup_quarantined`: "Automatic recovery stopped / operator recovery required" > - Restarts and runner loss are product failures. Interrupted work must recover by itself, and it must never get two writers in one workspace > - This pull request makes runnerd journal its process lineage to disk, and makes close stop and prove gone every journalled process after runnerd is lost > - The benefit is that a killed runnerd leads to an automatic, safe retry within the normal attempt budget, instead of an operator stop ## Linked Issues or Issue Description Refs #20259 (runner loss classification and the lineage sampler panic). Refs #20260 (automatic continuation of an inert cleanup quarantine). **What happened?** A chat eval killed only the owned local `runnerd` process (SIGKILL) during an OpenCode tool call (`chat-features.runner-opencode.local.recovery.retry-exhausted`, `recovery.explain-stall` and `recovery.source-action`). Attempt 1 recorded `native_runner_process_exited` with "Retry scheduled 1/3". Attempt 2 stopped with `native_session_cleanup_quarantined`. The task was blocked with no owner, and the next action was operator recovery. **Expected behavior** After runnerd is lost, the product stops the provider and tool processes that runnerd started, proves them gone, and continues the run within its retry budget. When it cannot prove that, it keeps the fail-closed quarantine. **Steps to reproduce** 1. Start a local native OpenCode (or Codex) run. 2. Let the agent run a tool that leaves the provider's session (for example `nohup job &`). 3. Send SIGKILL to the `runnerd` PID only. 4. Close rejects with `runner did not durably suspend before checkpoint`, and the next attempt fails with `native_session_cleanup_quarantined`. **Paperclip version or commit** master at 4c15eea770 ## What Changed - `process_lineage.rs`: runnerd journals each supervised process's lineage to `<runner state>/process-lineage/lineage-<runnerd pid>-<n>.json`. The journal holds the root, every member, and every group and session a member leads, each with its start time. The first record is written when the provider is spawned, before any tool can run. Later records are written by the existing 50 ms sampler only when the record changes. A teardown that proves the lineage gone removes the journal. - `process_lineage/journal.rs` (new module): the on-disk format, the write rules and the reaper `reap_journals`. The reaper refuses when runnerd never journalled (no writer marker) or when runnerd is still alive. It restores each journal into the same `LineageState` rules, sends TERM and then KILL only to processes whose PID and start time still match, and reports `verified` only when none remain. A journal that it cannot read refuses the proof. - Unsampled escapes: each record stores `sampled_through`, the newest process start that its read listed. The value is refreshed at most once a second while the membership is unchanged. The reaper refuses (`unattributed_orphan`) when a same-user orphan (parent PID 1) that is in no journal started after that point. Such a process can be a tool that started after the last read, so the reaper neither signals it nor ignores it. - A failed journal write removes the writer marker. Removal needs no free space. After that, no reap can verify from a record that may be missing a member. - `paperclip-runnerd`: durable mode enables the journal before any provider starts. A new `reap-lineage --state-dir --runner-pid` mode prints the report and exits 0 only when verified. - `runnerd-codex-transport.ts`: close samples whether runnerd exited unsuspended before close sent any signal. In that case, for a runnerd that this transport spawned on this host, it runs the reaper from the same executable. A verified report ends the session as a crash stop with its durable record intact. Recovery then continues the run. Any other result keeps the existing `NativeSessionCloseUnrecoverableError` quarantine. Launcher-owned (remote) and adopted runners do not use this path. - `runnerd-codex-transport.ts`: close does not queue a durable `runner.suspend` for a runnerd that it already knows is lost. Before, that suspend stayed pending in the saved control-plane state. A later runnerd on the same root then failed at startup with `terminal command result acknowledgement changed its durable identity`. The shared suspension barrier is unchanged for remote and adopted runners. - `runnerd-codex-transport.ts`: a runnerd that exits during close (for example because its provider snapshot is unreadable) is not a loss. It keeps the quarantine. - `runnerd-codex-transport.ts`: `#runnerHasExited` treated a signal-killed runnerd as alive, because `exitCode` is `null` and only `signalCode` is set. Close then spent its whole deadline trying to suspend a dead runner. It now checks `signalCode` too. - `fake-codex-app-server`: `--spawn-escaped-tool-job <pidfile>` starts a tool job in its own session at turn start. - `doc/execution-semantics.md`: describes the journal, the reaper and the fallback. ## Verification - Rust, `cargo test -p paperclip-runner-core --lib process_lineage` (14 pass) and `process_supervisor` (20 pass). New cases: - `reaping_refuses_an_orphan_born_after_the_last_sample`: a tool starts after the last read, starts a new session and exits, and leaves a non-leader orphan job. The reap refuses and does not signal the job. It fails without the guard. - `a_failed_journal_write_withdraws_the_writer_marker` and `an_unchanged_record_is_rewritten_only_to_advance_sampled_through`. - `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`: a real provider and a real `setsid` tool job, with a dead writer. The reap stops both, proves them gone and retires the journal and marker. - `reaping_refuses_without_a_marker_or_while_the_writer_lives` and `reaping_refuses_an_unreadable_journal`. - `a_journalled_record_restores_the_same_lineage`: after a round trip, the restored record still finds a re-parented job and its later child, and ignores an unrelated process. - TypeScript, live with the real debug runnerd: `stops a lost runnerd's provider and escaped tool job from its lineage journal`. - `journal=true`: SIGKILL runnerd mid-turn. The escaped tool job outlives runnerd. Close resolves, the job is gone, and the report is `verified`. Before the fix, close rejected with `runner did not durably suspend before checkpoint` and the job stayed alive. - `journal=false` (an older runnerd with no journal): close still rejects, and the job is not signalled. - `runner-lineage-reap.test.ts`: a report from an older runnerd, a malformed report, or an inconsistent `verified` report never counts as proof. - Turn-start cost: the journal adds one single-process read and one small file write when runnerd spawns its provider (about 1.5 ms in a macOS debug build), once for each runnerd. Nothing is added for each turn. - Paid runner E2E, `chat-features.runner-opencode.local.recovery.retry-exhausted`, on the eval branch with this fix (runs 42637596571 and 42644400434). Before the fix (run 42615758956), attempt 2 stopped with `native_session_cleanup_quarantined`. Now each of the three runnerd kills records `native_runner_process_exited`. Attempts 1 and 2 are `retryable_failure`, and attempt 3 is `terminal_failure` with cause `native_session_retry_exhausted` (attempt 3 of 3). The reaper ran after the first kill and verified. The cell still fails one eval setup check. The check waits for an *active* exhausted recovery action, but `execution-recovery-resolution.ts` settles that action as resolved and blocks the task (`preserve_without_replay_v1`). That is a separate grader decision, outside this PR. ## Risks - Medium. The change only affects a close where runnerd already exited without suspending and without a signal from close. Every other close is unchanged. - The reaper signals processes by PID. Each signal re-checks the PID's start time first, and a group or session is used only while its leader PID still names the recorded process (the same rules as the existing in-process teardown). - A crash stop leaves the durable root unsuspended. The coordinator already scheduled a retry for `native_runner_process_exited`. That retry now runs instead of failing on the quarantine. The paid E2E shows that attempts 2 and 3 start on the same run. - The escape guard can refuse when an unrelated same-user process becomes an orphan in the window of about one second after the last read. The fallback is today's quarantine. An orphan of a subreaper other than init is not seen by the guard. - The journal adds at most one small write each second for each runnerd while processes start on the host. This happens on the background sampler, not before turn start. - Previous-release story: no database or saved-session schema change. The journal is a new file in the runner state directory. A runnerd from an older release writes no journal and has no `reap-lineage` mode. Its loss keeps today's quarantine, and the existing board Retry and stopped-session proofs still apply. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:19:41 -06:00
DottaandClaude Opus 5.5	e015c2ef21	fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285) ... ## Thinking Path - #20145 prepares the next Claude provider generation at park, and #20190 restarts the ACPX provider when a turn settles. They move the sidecar respawn off the follow-up's critical path. - When that turn-end restart fails, the runner "fails closed": `restart_session_after_turn` saves the session `closed` and drops it. Within the run, the next turn fails exactly like a failed restart at turn start. That's intended. - Across runs it breaks the conversation: - the next run's `run.attach` refuses a `closed` session ("run.attach requires the same settled ACPX provider profile and session"); - it reconnects for about 10 s; - the controller then records a continuity break and starts a fresh Claude conversation (`run.startup.reuse provider_session: fresh/created`). - A local A/B (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`, on a host with load average about 17–27) confirmed it: | | d40a7494 (before #20145/#20190) | a3a25ce6 (after) | |---|---|---| | Follow-ups that resumed the conversation | 3 of 3 (`verified_reuse`, resume 4.4–8.2 s) | 1 of 3 | | Follow-ups that started a fresh conversation | 0 of 3 | 2 of 3 (bootstrap 16–17 s) | The kept instance shows each lost follow-up's runner received provider state `lifecycle: "closed"`, with `run.attach` refused. - A failed park preparation already recovers safely: it waits for the reaped provider's lifetime fence, then saves `suspended`. The turn-end restart should do the same. ## Linked Issues or Issue Description Refs #20190, #20145 **What happened?** When the post-turn ACPX provider restart failed, the next Claude run could not attach. It spent about 10 s and then started a new conversation, losing the task's conversation context. **Steps to reproduce** 1. Run a Claude task on a busy host so the turn-end provider restart fails (the fixture `reject-open` fault reproduces it). 2. Post a follow-up comment (a new run). 3. The follow-up's `run.attach` is refused, then `run.startup.reuse` reports `provider_session: fresh/created`. **Expected behavior** The follow-up resumes the same Claude conversation. Within the failing run, later turns still fail closed. ## What Changed - `acpx_park_preparation.rs`: - A failed `restart_session_after_turn` now calls the shared `recover_failed_generation_restart`, renamed from `recover_failed_park_preparation`. - That recovery drops the session, marks it `prepared` with the provider exit unconfirmed, waits for the provider-lifetime fence, then saves `suspended`. - If the fence or the save fails, the state stays exit-unconfirmed, which is the safe direction. - `acpx_provider_backend.rs`: - Later turns of the failing run still fail closed: the first with "ACPX turn start failed closed: <cause>", then "ACPX session is not open". That's the same as a failed restart at turn start. - The restriction is saved in the ACPX state as `turnEndRestartFailedRun` (serialized only while set), so it outlasts a Runner process restart inside the same run. Same-run restore doesn't reopen the session while it's set. - Only a `run.attach` from a different run clears it; a same-run attach keeps it. State validation requires it to match the descriptor's run. - The cause is handed to the durable Runner once (`take_turn_end_restart_failure`), which records it as a redacted `turn-end provider restart failed: <cause>` diagnostic in `runner-state.json`. That gives the under-load cause a first recorded sample. - Tests (`tests/support/acpx_turn_end_restart.rs`): - New: `the_next_run_resumes_the_conversation_after_a_failed_turn_end_restart`, red on master (the state is `closed`). It covers the same Runner process with a warm attach, and a new Runner process with a cold attach. The conversation id is preserved, and the next run completes a turn. - New: `a_runner_restart_in_the_same_run_keeps_the_failed_turn_end_restart_closed`, red without the saved field. A new Runner process with the same run id fails `turn.start` without spawning a provider; then a new run cold-attaches and resumes the same conversation. - The existing `a_failed_turn_end_restart_fails_the_next_turn_like_a_failed_turn_start_restart` now also asserts that the turn-end failure saves `suspended` (the turn-start failure stays `closed`). Everything else about the run's failing turns is unchanged. ## Verification - `cargo test -p paperclip-runner-core`: - `--test acpx_park_preparation`: 21 passed. - `--lib acpx`: 93 passed. - `acpx_park_readiness`: 8 passed. - `acpx_warm_attachment`: 4 passed. - `acpx_provider_suspend`: 5 passed. - `acpx_provider_session`: 12 passed. - `acpx_provider_turns`: 29 passed. - `native_provider_backend`: 47 passed. - `durable_recovery`: 3 passed. - Live A/B on this branch (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`): follow-ups resumed 3/3 (1/3 before), follow-up first activity p50 6.2 s (about 20–23 s before). - `cargo clippy --all-targets`: no findings on changed lines. ## Risks - **Late-callback guarantee:** unchanged. The failed restart already reaped the old generation, and the next generation starts only via `run.attach` or a later reopen. That's the same resume path a failed park preparation uses. - **Fence wait:** the failure path now waits for the reaped provider's lifetime fence before saving, matching park recovery. If the fence never clears, the state stays exit-unconfirmed and fails closed. - **Rollback:** `AcpxDurableState` denies unknown fields. The new field is written only between a failed turn-end restart and the next run's attach. A rollback inside that window makes the older Runner refuse the saved state, which still fails closed, as master's `closed` state does today. - **Scope:** the cause of restart failures under load isn't changed here. This makes their failure mode safe. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:15:25 -06:00
Nicky LeachandPaperclip	09412e652a	fix(runner): allow warm release to drain after provider restart (#20290) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm Runner sessions park after a turn so follow-up tasks can claim the same process. > - Claude restarts its ACPX provider after the controller receives the terminal result. > - That restart can delay the drain receipt beyond the release path's one-second limit. > - A failed drain prevents parking, so the idle timer later closes the retained Runner. > - This pull request uses the existing warm admission budget for release drain. > - The benefit is retained Runner reuse without moving provider restart onto task dispatch. ## Linked Issues or Issue Description Fixes: #20225. Refs #20190 and #20145. Related performance follow-up: #20224. ## What Changed - Align the release drain timeout with the existing 30-second warm admission budget for every provider. Close paths keep their explicit remaining deadlines. - Keep the existing immediate completion when the drain receipt proves settlement. Add no sleep, retry, provider branch or dispatch work. - Add a real Runner regression with a fake ACPX sidecar whose replacement start takes 1.5 seconds. One release call must park the Runner, then a follow-up must claim the same Runner process. - Document the release drain budget. ## Verification - Targeted command in `packages/paperclip-runner`: `vitest run src/live/runnerd-codex-transport-release-resume.test.ts src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed** across 3 files, 157.78 seconds. - Independent parent verification reran the slow-restart and ordinary cached cases: **2 passed**. Source review confirmed that release is the only caller using the changed default; close callers supply explicit deadlines. - Negative control: with the original one-second timeout, the new test fails because the single release call returns `false`. The fixed code passes the new case. - Built only the current-source Runner and two fixture binaries required by these tests. The first sandboxed invocation could not bind its loopback listener (`EPERM`); the test ran with loopback access afterward. - Full GitHub PR workflow [42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993) passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard Greptile scored this same head 5/5 with no review threads. Full repository checks ran in GitHub CI, per the requested local test scope. No live Claude session or hosted deployment was tested. ## Risks - An unresponsive Runner can now hold release drain for up to the existing 30-second warm admission limit. A healthy Runner returns as soon as its receipt arrives. The provider restart stays at turn end; task dispatch gets no fixed delay. - Saved state, protocol and provider identity formats do not change. Previous-release state carries forward under the existing validation. - The controlled fake-sidecar test proves the restart timing race and process reuse. It does not qualify live provider latency or a hosted rollout. ## Model Used - OpenAI GPT-6 through Codex, with repository tools, code editing and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 07:43:27 -08:00
DottaandClaude Opus 5.5	1b09920a64	fix(issues): refuse agent status changes that detach another run's live execution (#20271) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that an agent run executes holds a checkout lock and an execution lock (`checkoutRunId`, `executionRunId`) > - A status change away from `in_progress` clears these locks. It does not stop the run that holds them > - Agents cannot stop another agent's run: run cancel and the task-tree pause are board-only > - But an agent that can edit a task (for example a lead with the manage-active-checkouts grant over its report) can change its status. The run then keeps running and is no longer bound to its task > - In chat evals, a lead "paused" a launch tree. It moved running subtasks to backlog. The runs kept running, detached > - This pull request refuses an agent's status change when a different queued or running run holds the task's lock. The denial is final, links the task, and names the person's controls > - The benefit is that agents cannot interrupt another agent's run through a status edit, and runs are not left orphaned ## Linked Issues or Issue Description Refs #20261 (chat guidance sends a task-tree pause to the person's Pause subtree control). Refs #19394 (resume restarts the work a subtree pause interrupted). **What happened** An agent changed the status of a task while another agent's run executed it. The request succeeded. The server cleared `checkoutRunId` and `executionRunId`. The run continued, but it was no longer bound to the task. In the chat evals `steering.stop-versus-pause` and `task-coordination.pause-tree`, a lead moved running subtasks to backlog to "pause" a launch. Their runs kept running, detached. **Expected behavior** An agent cannot get the effect of interrupting another agent's run. When a different live run holds the task's lock, the agent's status change is refused before any write. The refusal says it is final. It sends the person to the task and names the controls that really stop or hold the work: the run's Stop or Interrupt, or More task actions, then Pause subtree on the parent task. The lock-holding run, board users, and paths that stop the run themselves continue to work. **Steps to reproduce** 1. Create agent A and agent B, with B reporting to A (A gets `tasks:manage_active_checkouts` through the manager chain). 2. Give B an `in_progress` task. Set `checkoutRunId` and `executionRunId` to a `running` heartbeat run of B. 3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with `{ "status": "backlog" }`. 4. Before this change: `200`, the status is `backlog`, both lock fields are `null`, and B's run is still `running`. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it refuses a status change when `executionRunId` or `checkoutRunId` names a different run in the same company that is `queued` or `running`. It runs in `PATCH /issues/:id` after the final status and assignee are known (execution-policy transitions included) and before any run stop or write. It also runs in recovery resolution when the source status changes. - The guard applies to any status other than `in_progress`, also to the current status, because the service clears the locks for those too. - The write is conditional on the lock the guard checked. The route re-reads the lock and passes it to `svc.update` as `expectedExecutionLock`. The service compares it under the row lock and returns `409 issue_execution_lock_changed` if a checkout landed in between. - These cases stay allowed: the lock-holding run itself, board users, requests with no status, a change to `in_progress`, `cancelled` (the route cancels the run), reassignment (the route stops the run first), `done` through PATCH while a Runner goal is active (terminalization pauses the goal and stops the run; recovery resolution does not get this exception), and a lock holder that already finished. - `packages/shared/src/issue-write-denial.ts`: new denial code `issue_status_change_live_run` (409). The copy says the refusal is final, links the task as `[identifier](/prefix/issues/identifier)`, and names Stop or Interrupt and More task actions, then Pause subtree. A new optional `issuePath` context field carries the link. The UI notice renders it through the existing generic path. - `doc/SPEC-implementation.md`: documents the rule in the checkout contract section. - Paths checked with no change needed: there is no bulk issue update route; tree holds and run cancel are board-only; `POST /issues/:id/release` already lets only the assignee's checkout run release; API-catalog actions call the same PATCH route; MCP status changes already refuse with `MCP_ACTIVE_EXECUTION`. ## Verification - Red on master, green with this change: - `server/src/__tests__/issue-stale-execution-lock-routes.test.ts` (embedded Postgres, real issue service): a manager moves its report's running task to backlog. On master: `200`, status `backlog`, `checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409 issue_status_change_live_run`, row unchanged, run still `running`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued lock holder, and a repeated current status (red on master). Allowed cases: field edits, the lock-holding run, board, `cancelled` (run is cancelled), and finished lock holders; an allowed change carries `expectedExecutionLock`. - Same embedded-Postgres file: `svc.update` with an `expectedExecutionLock` that differs from the row returns `409 issue_execution_lock_changed` and leaves the row unchanged. - Also passed locally: `issue-closed-workspace-routes`, `issue-comment-reopen-routes`, `issue-recovery-actions`, `issue-scheduled-retry-routes`, `low-trust-red-team-routes`, `issue-update-comment-wakeup-routes`, `external-object-routes`, `packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`. - `tsc --noEmit` shows no errors in the changed files. Full typecheck, build and test are left to CI. ## Risks - Behavior change for agents: a lead or CEO that used status edits to take over a report's running task now gets a 409. The supported paths (comment, cancel, reassign, or the person's Stop or Pause subtree) still work. - A queued run that holds the lock also blocks the change. This is deliberate: the queued run would otherwise start later with no task bound to it. - Agents can still cancel another agent's run by cancelling its task. That is existing, documented behavior ("task cancellation cancels the intent"), and this PR does not change it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:21:40 -06:00
DottaandClaude Opus 5.5	3374ca54f5	fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 06:38:24 -06:00
DottaandClaude Opus 5.5	8eb11fa8aa	fix(runner): fail closed with a recorded cause when runner event commits stall (#20235) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Runner streams numbered events to the control plane, which commits each one and acknowledges the committed prefix > - In restart chaos wave `w-20261109T093417-s9807`, t01 (claude R1, local), the control plane never committed Runner seq 34. Every failure path in `#event` closed the connection without recording why. The Runner reconnected 2,848 times over 14 minutes and nothing detected it > - This pull request makes that loop a detected, named failure that carries its cause > - The benefit is that a wedged event stream becomes a visible run failure with a reason, instead of a silent "running" task ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T093417-s9807` (master `df4562ab38`), task t01, run `d08dc899`. **What happened** (Phase 2's re-check of the kept instance): - The provider kept working. Claude's transcript runs to 09:47:03 and ends in `paperclip_finish`. - The Runner, its ACPX provider state, and the controller journal agree on acked seq 33. The controller never commits Runner seq 34 (Write progressed, 09:36:25.892). - Phase 2 replayed the kept outbox offline. Runner seq 34 fails `validatePrpEvent`: its `/payload/target` contains a newline from a malformed Claude Write call. `#event` closes without a reason, and the Runner redelivers the same bytes forever. (Phase 2's separate PR stops the producers from emitting such paths.) - Every `#event` failure path closed the connection without recording a reason, so the cause was invisible. An outbox that already holds such an event can only be cleared by failing closed. - The `<instance>:<run>:N` ids in `heartbeat_run_events` come from the TS driver's own projection numbering, so there was no Runner sequence rewind. ## What Changed - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: - `#eventCommitFailed` replaces the bare `connection.close()` on each `#event` commit-failure path. It records a bounded cause: `event_sequence_mismatch` for an out-of-order event, `event_schema_invalid:<issue paths>` for an event that fails `validatePrpEvent`, the error's code or name (never its message) for a commit callback error, `journal_sequence_not_contiguous`, `journal_commit_refused`, or `checkpoint_flush_failed:<cause>`. - If failures continue for the same run at the same acknowledged cursor for at least 20 attempts and 45 s, it fails closed with `NativeSessionProtocolIntegrityError("runner_event_commit_stalled", "<cause> at source seq N")`, where N is the failing event's own sequence. Only a moved acknowledged cursor counts as progress (a replayed, already acknowledged duplicate does not), and a different run starts a fresh counter. A single transient failure still just reconnects, as before. - A checkpoint flush that fails before any outbound frame also counts, as `checkpoint_flush_failed:<cause>`. That includes the authentication challenge and the welcome, so a reconnect that cannot even authenticate still reaches the bound. - A journal write that throws (`#store.commit`, `#store.commitEvent`, or the welcome save) leaves the store unwritable until reload, so it fails closed at once with `journal_write_failed:<code>`. - `packages/paperclip-runner/src/contracts/native-session-backend.ts`: new reason `runner_event_commit_stalled`, with an optional `detail` in the message. ## Verification Red/green: - "bounds commit-stall reconnects for an out-of-order event": a schema-valid event with the wrong `sourceSeq` fails closed with `event_sequence_mismatch at source seq 5`. Before this it closed silently forever. "… for 20 failures inside 45 s" pins the time bound: 20 failures over 19 s do not fail closed. - "fails closed when checkpoint storage keeps failing before a reconnect authenticates": with checkpoint persistence refusing (`ECONNREFUSED`), 20 reconnect attempts fail before authentication, then the control plane fails closed with `checkpoint_flush_failed:ECONNREFUSED at source seq 1`. Without this, those attempts close silently. - "fails closed at once when the journal cannot record a runner event": a journal write throwing `ENOSPC` fails closed with `journal_write_failed:ENOSPC at source seq 1`, and the private error text is not carried. On master the connection only closes. - `durable-prp-control-plane.test.ts` "fails closed with the schema issue when the runner keeps redelivering an invalid event (restart chaos t01)": an event that fails `validatePrpEvent` is redelivered 20 times over 57 s of mocked clock. On master the control plane only closes; with this PR it fails closed with `runner_event_commit_stalled` and detail `event_schema_invalid:/… at source seq 1`. - `durable-prp-control-plane.test.ts` "fails closed when runner events keep failing to commit without progress": 20 reconnects over 57 s of mocked clock, each commit failing with `ECONNRESET`. On master nothing fails closed. With this PR the 20th failure reports `runner_event_commit_stalled` with detail `ECONNRESET at source seq 1`, and the private error message is not carried. File: 201 passed, including the existing "keeps a non-typed commit failure retryable" cases. Static checks: - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. The server is unchanged. ## Risks - Storage that keeps failing event commits for at least 45 s and 20 attempts now fails the run (operator recovery) instead of looping silently. Shorter outages are unchanged. - The recorded cause is a code or error name only, never error text, so no tool content or credentials reach the run error. - No schema change and no change to persisted state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:57:46 -06:00
DottaandClaude Opus 5.5	3f2a940d5b	fix(runner): answer held Paperclip tool calls after a local restart before the next turn (#20227) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart interrupts a turn, the Runner keeps each Paperclip tool call the model made as "held" until Paperclip answers it, and it refuses to start the next turn while any call is held > - Paperclip answers held calls after a reattach through its tool recovery path. That path was wired only for remote runners > - In restart chaos wave `w-20261109T084023-s9403`, t101 (local codex, K2 then K1) held a `read_document` call that nothing answered, so the continuation's `turn.start` failed and the task needed a Retry > - This pull request wires held-call recovery for local runners too, and makes `turn.start` answer or wait for held calls first > - The benefit is that a turn lost to a restart no longer strands the run behind one unanswered call ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T084023-s9403` (master `fc5c79a327`), task t101 (codex C1, K2, local). **What happened:** - Run `4b5810d5` called `read_document` (`exec-ac8fa9b4`) around the K2 drain at 09:08:18. A K1 then killed the next server start. - At 09:08:36 the third controller reattached. The Runner restored the conversation and reported the old turn lost (`provider_turn_lost_on_restore`, recoverable). The held call stayed pending, by design: a held call accepts a late result. - The controller had the call: the quarantine detail lists `exec-ac8fa9b4` under `semanticTools.pending`. But `recoverPendingSemanticInputs()` returned at once: for a local run `controllerPersistence` is unset, so `recoverSemanticToolInput` was `undefined`. - At 09:08:46 the continuation's `turn.start` failed with "cannot prepare the next provider turn while tool calls are active (pending: 1 [read_document:exec-ac8fa9b4…])". The run ended `native_session_interrupted`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: the runnerd transport now always gets `recoverSemanticToolInput`, not only when `controllerPersistence` (remote) is set. It uses the same epoch check and `SessionAuthorityEpoch.recover`. Recovery re-authorizes and re-runs reads, and replays receipts for mutations; it never repeats an uncertain mutation. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts` (`#startTurnWork`): when the control plane has unsettled semantic inputs, it runs `recoverPendingSemanticInputs()` before queueing `turn.start`, and waits for that pass, bounded by the turn-start deadline. Recovered results are queued as commands ahead of `turn.start`, and the Runner applies commands in order. If recovery has not finished by the deadline, the turn start fails with `native_turn_start_held_tool_calls_unresolved` instead of sending `turn.start` behind it. When everything is settled there is no wait, so a normal turn start is unchanged. Live calls of a turn being replaced are not waited on; the replaced start still rejects them. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: `recoverPendingSemanticInputs()` passes are serialized. A call made while the startup pass is still answering a held call waits for that pass, then runs its own, so the turn-start gate cannot finish early by skipping a call that is already being recovered. ## Verification Red/green: - `native-session-executor.test.ts` "answers a local run's held Paperclip tool calls after a restart (restart chaos t101)". On master `recoverSemanticToolInput` is `undefined` for a local run; with this PR it is wired. - `runnerd-codex-transport.test.ts` "answers a held call before starting the next turn (restart chaos t101)", against real runnerd and fake Codex, with a slow startup recovery pass already in flight when `turn/start` arrives. On master `turn/start` never ran recovery, and without the serialization `turn.start` is queued before the held call's result. With this PR the held call is recovered once and its result is queued before `turn.start`. Regression and static checks: - `runnerd-codex-transport.test.ts`, `durable-prp-control-plane.test.ts`, `runnerd-codex-transport-adoption.test.ts`: 510 passed. - `native-session-executor.test.ts` and `native-session-resume.test.ts`: 1368 passed, 1 failed. The failing case, "archives a damaged prior epoch … with real runnerd", fails identically on master on this Mac (`ENOENT … authority-epochs`), so CI is its reference. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Local runs now recover held calls after a reattach, as remote runs already did. Reads re-run under current authority. A mutation without a saved receipt still returns no result and stays held, as before. - `turn.start` may now wait up to its existing deadline while a recovery pass runs, then fails clearly if it has not finished. It never waits when nothing is unsettled, so normal turns get no added latency before `turn.start`. - No schema change and no change to persisted run state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:26:12 -06:00
DottaandClaude Opus 5.5	465da4227a	fix(runner): drop a tool target path that breaks a line so the controller can acknowledge its event (#20238) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner turns provider tool calls into durable PRP events. The controller acknowledges each event only after it validates and commits it. > - The controller's event schema forbids line terminators in a safe path such as `tool.execution` `target`. > - The ACPX sidecar and the Rust runner accepted a location path that contained a newline. Claude sometimes emits a malformed Write call that puts the file content into the path. > - The controller refused that event without an acknowledgement. The runner redelivered the same bytes, and the run stopped making progress for 14 minutes (2,848 reconnects). > - This pull request makes both producers drop such a path, so the event carries a null target and the controller accepts it. > - The benefit is that a malformed provider path can no longer wedge a run's event delivery. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave seed 9807 (claude, R1, local), the run stopped receiving events after runner source seq 33. The runner reconnected 2,848 times in 14 minutes. Offline replay of the saved runner outbox through the controller's `validatePrpEvent` showed that seq 34 fails `/payload/target` (the safePath pattern), because the target is `packages/shared/src/chaos/slug-2c5c3c8f.ts\n<parameter name="content">export function chaosSlug…`. Seq 35, 39 and 40 (the same tool call) fail the same way. **Expected behavior** The runner never emits an event that the controller's schema refuses. A malformed tool path becomes a null target, and the tool event still arrives. **Steps to reproduce** 1. Have an ACPX provider send a tool call whose location path contains a newline (for example, a Write call with the content in the path). 2. See that the runner emits `tool.execution.progressed` with that target. 3. See that the controller closes the connection without an acknowledgement on every redelivery. **Paperclip version or commit** master at 3963d38560 ## What Changed - `runner-core/src/tool_file_changes.rs`: new `is_single_line_path`, which refuses control characters and U+2028/U+2029 (the characters ECMAScript `.` does not match). `normalize_relative` uses it, so file-change paths are covered too. - `runner-core/src/provider_events.rs`: `safe_acpx_location` uses the same check (it replaces the NUL-only check). - `src/drivers/acpx/safe-locations.ts`: the sidecar drops a location whose path contains a control character or U+2028/U+2029. - Tests: a Rust test covers the observed path, `\r`, U+2028, U+2029 and tab, plus a full ACPX tool-call normalization with a broken location and file change. A TS sidecar test covers the same paths. The controller-side detection and reason capture for a refused event is in paperclipai/paperclip#20235 (Phase 5). This PR does not touch the controller. ## Verification - Red before the fix: the Rust test `drops_a_location_whose_path_breaks_a_line` and the TS test `drops a path that breaks a line…` both failed. Both pass after it. - `npx vitest run src/cli/acpx-sidecar-locations.test.ts src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx` in `packages/paperclip-runner`: 1,818 passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `packages/paperclip-runner`: passes. - `cargo test --lib` in `runner-core`: 598 passed. Four `task_memory::tests::monitor_*` tests fail on this macOS host in isolation too. That file is unchanged here (last changed in #18801), and the CI Rust job is the authority. ## Risks - Low. Only a path that the controller already refuses is dropped. A valid POSIX filename with a tab is now dropped as a display target too, which is consistent with the existing ACPX rich-event check (`char::is_control`). - Previous-release story: a runner outbox already saved with such an event (from an older runner) still holds it. That case needs the controller detector in paperclipai/paperclip#20235, which fails the run closed instead of looping. No saved state, schema or pack contract changes here. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:06:43 -06:00
DottaandClaude Opus 5.5	d2fe85e57c	fix(secrets): delete a secret that a binding proposal referenced (#20193) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose a secret binding. A person approves or rejects it on a review card. > - Each binding proposal row refers to one source: a secret or a pending secret proposal. A check constraint enforces this. > - The secret reference used `ON DELETE SET NULL`. When a person deletes the secret, Postgres sets the reference to null. The check constraint then rejects the row. > - So a person cannot delete any secret that a binding proposal ever referenced. The API returns 500. > - This pull request deletes the proposal rows together with the secret, as the other source reference already does. Before the delete, it rejects pending proposals so their cards close with a reason. > - The benefit is that secret deletion works again and no review card points at a removed proposal. ## Linked Issues or Issue Description Refs #15644 (related: block secret deletion while bindings still exist). **What happened** `DELETE /api/secrets/:id` returned 500 for a secret that a binding proposal referenced. The server log showed `new row for relation "company_secret_proposals" violates check constraint "company_secret_proposals_shape_check"`. This happened for pending, rejected and approved proposals. It also happened after the person removed the agent's binding. **Expected behavior** The person can delete the secret. Proposals that referenced it go with it. A pending review card closes as rejected and tells why. **Steps to reproduce** 1. An agent proposes a binding for an existing company secret (`POST /api/agents/me/secret-proposals` with `secretId`). 2. Approve or reject the proposal, or leave it pending. 3. Delete the secret with `DELETE /api/secrets/:id`. 4. The response is 500 and the secret stays. ## What Changed - `packages/db`: `company_secret_proposals.secret_id` now uses `ON DELETE CASCADE`, the same as `secret_proposal_id`. Migration `0452_secret_proposal_secret_cascade.sql` replaces the foreign key (`DROP CONSTRAINT IF EXISTS`, then `ADD CONSTRAINT`). - `server/src/services/secrets.ts`: `remove` accepts an optional `beforeDelete` hook. The hook runs after the secret is marked deleted and before its row is removed. - `server/src/services/secret-proposals.ts`: new `rejectPendingBindingsForSecret`. It rejects pending binding proposals for a secret with a reason and returns them. This closes their review cards and writes the normal activity entries. `createBinding` now holds the secret row `FOR SHARE` in its insert transaction and refuses a deleted secret. So no new pending proposal can start after the delete marks the secret deleted. - `server/src/routes/secrets.ts`: the delete route rejects pending proposals in `beforeDelete`. For each one it sends the normal rejection notice and wake (`notifySecretProposalResolution`), so the proposing agent continues. - `server/src/__tests__/secret-proposals-routes.test.ts`: a regression test deletes a secret with a pending, rejected and approved binding proposal through the route. It checks the secret and proposal are gone, the card status is correct, and only the pending case sends one rejection notice and one wake. A second test checks that a binding proposal for a secret that the delete already marked deleted fails with 404. ## Verification - Red first: the new test failed on `origin/master` with the shape-check violation for all three cases. - `cd server && vitest run --config vitest.config.ts src/__tests__/secret-proposals-routes.test.ts`: 47 passed. - `vitest run src/__tests__/chat-run-milestone-governed-work.test.ts src/__tests__/agent-secrets-routes.test.ts src/__tests__/secrets-service.test.ts`: 112 passed. - `tsc --noEmit` for `server` and `packages/db`: pass. `pnpm --filter @paperclipai/db run check:migrations`: pass. - Found by the `secrets.secret-proposal` chat evaluation. Its cleanup deletes the fixture secret and failed on this error in runs 42563093640 and 42571127911. ## Risks - The migration only replaces one foreign key. It does not rewrite rows. Existing rows already have a non-null `secret_id` or `secret_proposal_id`, so nothing else changes. - Deleting a secret now also deletes its binding proposal rows. The activity log keeps the proposal history (`secret.proposal.*` and `secret.deleted`). Secret-kind proposal rows that created the secret keep `created_secret_id ON DELETE SET NULL`, so their history stays. - Other paths that delete secrets (company deletion, user secret cleanup) now cascade instead of failing on the same constraint. They do not reject pending cards first. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:54:07 -06:00
DottaandClaude Opus 5.5	d7d0f96f3f	perf(runner): save provider state once per poll instead of once per frame (#20217) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode, Codex and Cursor Cloud chat turns run through runnerd. runnerd reads provider frames in order and keeps the run's provider state in a durable file > - For every frame that it reads, runnerd rewrote the whole provider state file and called `fsync` on the file and on its directory. The state holds every tool result kept in the turn, so each save got more expensive as the turn went on > - A tool call request is just the next frame. It waited behind one full save for every frame before it. #20209 measured tool-call turnaround that grew from about 1.4 s to about 15 s in one turn > - #20209 stopped the OpenCode proxy from sending frames that runnerd discards. Every frame that runnerd still reads still paid a full save, and other proxies (for example Cursor Cloud) still send such frames > - This pull request makes runnerd save the frames of one poll once, before anything derived from them leaves runnerd. A save that would write the bytes already on disk is skipped > - The benefit is that the number of saves before a tool call no longer grows with the number of streamed frames. In a local measurement, a tool call after 400 frames with 830 KB of kept state took 0.57 s instead of more than 30 s ## Linked Issues or Issue Description Refs #20209 Related: #17514 (the same per-event full-state rewrite, on the server control-plane side) **What happened** During a native OpenCode chat turn, tool-call turnaround grew from about 1.4 s to about 15 s as the turn went on. runnerd saved its complete durable provider state, with `fsync`, after every provider frame. That state includes the tool results kept so far in the turn. A tool call request waited behind one full save for each earlier frame. **Expected behavior** The time to deliver a tool call must not grow with the number of assistant-text, reasoning or item frames that came before it. A crash or restart must still never lose state that a resumed run needs. **Steps to reproduce** 1. Start a native OpenCode (or Codex app-server or Cursor Cloud) turn that keeps a few tool results and then streams many text or reasoning frames before its next tool call. 2. Measure the time from the tool call frame to the `semantic_tool.input` event. 3. See that the time grows with the number of earlier frames and with the size of the kept state. Locally, `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture` on `master` reports 102 saves and 9.3 s for 100 frames with 830 KB of kept state, and more than 30 s for 400 frames. ## What Changed - `packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs` and `codex_provider.rs`: - The provider frame loop moves into `poll_provider_frames`. An ordinary notification frame and a descendant notification frame no longer save the state. They mark it as unsaved. - After the loop, `poll_current_provider_with_control` saves the unsaved state once. It also saves when a frame fails, before it returns the error. - Frames that already saved at once still do so: turn terminals, Cursor Cloud terminal deliveries, `reconciliation_required`, protocol failures, MCP permission and runtime requests, process exit, and new tool calls. - A tool call that runnerd answers at once (an exact completed replay or the receipt-limit stop) saves unsaved state before it writes to the provider. A rejected tool call already saved first. - `retained_events`, `execute` and `host_park_ready` save unsaved state first, so no event, command result or park decision can expose state that is not on disk. - Courtesy rejections that `CodexProvider::poll` writes by itself (unsupported requests, descendant requests, delayed requests after a terminal, full runtime-request capacity) are held while runnerd reads a batch (`hold_courtesy_responses`). runnerd sends them only after it saves the state that covers every earlier frame (`save_before_provider_writes`). A terminal frame also saves and releases them before runnerd asks the provider for goal state or drains the accounting tail. Released courtesy replies are best-effort, as courtesy replies were before: a provider that already closed stdin cannot turn a received terminal into a polling failure. The receipt-limit stop sends held cancellations before its `turn/interrupt` request, so their order to the provider does not change. - The trace records for runnerd's normalization (`rust_durable_normalization`), which name emitted event ids, are kept until the state that holds those ids is saved. The trace therefore only names saved events. Saved mappings are written before an exited or failed provider is dropped, so they are not lost. Raw frame records still go to the trace when a frame is read; they name no event ids. - `save_unsaved_frame_state` does not write when the serialized state is byte-for-byte the same as the saved file. All other saves write as before. - Serialization moves to `serialize_provider_state`, and the file write moves to `write_state_bytes`. The write path (private temporary file, `fsync`, rename, directory `fsync`, `0600`) does not change. - New tests in `packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs`: - `streamed_frames_do_not_multiply_durable_saves_before_a_tool_call`: 8 and 48 streamed frame pairs before a tool call need the same number of saves (1). On `master` they need 18 and 98. - `batched_frame_state_is_saved_before_events_leave_and_survives_a_restart`: at every poll, the saved file is equal to the in-memory state and holds exactly the delivered events. A new executor loads the file after the old one stops and gets the same state, the same unacknowledged events and the pending tool call. - `batched_frame_state_is_saved_when_a_later_frame_fails`: a frame that fails after 20 batched frames still leaves those 20 frames on disk. - `previous_release_state_is_rewritten_by_the_first_batched_save`: a state file in an older shape (compact JSON without newer default fields) loads, and the first batched save rewrites it. Unchanged state is not rewritten. Changed state is. - `courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved`: when the courtesy rejection reaches the provider, the saved file (what a restarted runnerd would load) already holds the 20 earlier frames and equals the in-memory state. It fails if the hold is removed. - `trace_interpretations_name_only_saved_events`: every event id that a trace record names is in saved state at the moment the record is written. It fails if records are written before the save. - `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`: after 122 frames are read and not yet saved, the saved file is byte-for-byte unchanged and runnerd has written nothing to the provider. A reload gets the pre-batch event sequence and no pending events. - `a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`: the provider closes stdin and then completes the turn with a tool call still pending. The held cancellation cannot be written, and the turn still completes and is saved. - `trace_interpretations_survive_a_provider_exit_in_the_same_poll`: the trace keeps the mappings for frames that arrived in the same poll as the provider exit. - `measure_streamed_frame_cost_before_a_tool_call` (ignored): prints saves and tool-call latency for different frame counts and kept-state sizes. ## Verification - `cargo test --locked -p paperclip-runner-core` (debug, macOS arm64): library 595 passed, 1 failed; all 35 integration test binaries passed (`--no-fail-fast`). The one library failure is `task_memory::tests::monitor_bounds_notices`. It also fails when run alone, takes about 9 minutes, and waits on file-watch timing in `task_memory.rs`, which this PR does not touch. In one earlier run, two ACPX park tests in `acpx_park_preparation` failed. The same binary also failed on unchanged `master` (with a different test), and it passed in the final run. - `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture`, debug build, macOS APFS (`sync_all` is `F_FULLFSYNC`). Each row is one turn: frames streamed, then one tool call. | Kept state | Frames | Saves (master) | Tool-call latency (master) | Saves (this PR) | Tool-call latency (this PR) | | --- | --- | --- | --- | --- | --- | | 4 KB | 100 | 102 | 946 ms | 1 | 18 ms | | 4 KB | 400 | 405 | 3,791 ms | 7 | 70 ms | | 211 KB | 100 | 102 | 3,207 ms | 1 | 37 ms | | 211 KB | 400 | 405 | 12,803 ms | 7 | 193 ms | | 833 KB | 100 | 102 | 9,251 ms | 1 | 93 ms | | 833 KB | 400 | about 405 | more than 30 s (test deadline) | 7 | 567 ms | With 400 frames, the remaining saves are one per poll of up to 128 frames and one per controller acknowledgement. - With the debug `paperclip-runnerd` and `fake-codex-app-server` binaries from this branch: `npx vitest run` on `src/live/runnerd-opencode-streaming-tools`, `runnerd-opencode-assigned-tools`, `runnerd-codex-transport`, `runnerd-codex-transport-settlement`, `runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`, `runnerd-saved-continuation`, `runnerd-detach-durability`, and `src/cli/opencode-proxy-output`, `opencode-proxy-events`, `opencode-app-server-proxy`: 11 files, 353 tests passed. ## Risks - Durability. The rule is: nothing derived from a frame leaves runnerd before the state that holds that frame is on disk. That covers delivered events, command results, park decisions, tool results and rejections, courtesy rejections, Cursor Cloud terminal acknowledgements, and trace records that name event ids. The tests above check each of these. - The unsaved window is at most one poll (128 frames). Every frame in it is undelivered: no event from it reached the controller, and runnerd wrote nothing to the provider while it was unsaved. `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged` proves this. A crash in the window therefore leaves the saved state and everything the controller and provider have seen exactly as a crash just before those frames were read. The resumed run starts from the same event sequence, so it cannot deliver an event id twice. Its normal recovery (thread resume and `thread/read` reconciliation) handles those frames in the same way as frames still in the provider pipe. On `master`, a crash could already lose the one frame being processed. - One provider write can still happen in the window: `thread/read`, which `CodexProvider::poll` sends to check the lineage of an unknown helper thread. It is a read-only query that waits for its answer, and it changes nothing on the provider side. - Courtesy rejections now reach the provider at the end of the poll instead of during it. The delay is one batch. The provider gets the same replies. - Previous-release story: the saved file shape does not change, and no new field is saved. A file from the previous release loads as before. The skip check compares exact bytes, so an older file is never treated as already saved. The first save after an upgrade rewrites it in the current shape. A deploy that lands mid-turn sees a file that the previous release saved after each frame. That file is valid input for this code. - The ACPX backend has its own frame loop and is not changed here. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file editing) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:43:27 -06:00
DottaandClaude Opus 5.5	088bdf7cfa	fix(daytona): wait out a sandbox state change before starting it for workspace realize (#20212) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, every run realizes its workspace in the leased sandbox. Realizing it first makes sure the sandbox is started > - Right after a control-plane restart, the sandbox can still be mid-transition, for example stopping or starting from the previous lifecycle step > - The plugin called `sandbox.start` straight away. Daytona rejects that with "Sandbox state change in progress", so the run failed `setup_failed` and needed a Retry > - This pull request waits out a sandbox's state change before starting it, and retries a start the provider rejected for that reason, within the existing start deadline > - The benefit is that a run whose sandbox is briefly transitioning starts on its own ## Linked Issues or Issue Description Refs #20211 **What happened?** Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503 (revision 9046884811), t07 (claude A1): run `5aabeb55` failed at 07:45:27 with `setup_failed` and "Failed to realize workspace for environment … (sandbox): Sandbox state change in progress". The sandbox was mid-transition after the restart, and realize neither waited nor retried. A Retry succeeded. **Expected behavior** Realize waits for a transitional sandbox state to settle, then starts the sandbox. If the provider still reports the state change, it retries, all within the existing start deadline. Any other start error fails at once, and a transition that outlasts the deadline still fails. **Steps to reproduce** 1. Realize a workspace while the Daytona sandbox is `stopping` or `starting`, or while `start` returns "Sandbox state change in progress". 2. Before this change, realize fails at once with that error. **Paperclip version or commit** 9046884811 (wave). Branch rebased on current master. ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `ensureSandboxStarted` now polls `refreshData` while the sandbox is in a transitional state. - The transitional states are `starting`, `stopping`, `restoring`, `archiving`, `creating`, `pending_build`, `pulling_snapshot`, `building_snapshot` and `resizing`. - It polls every second, until the start deadline from `timeoutMs`. Each read keeps the existing liveness bound. - If the start fails with "state change in progress", it refreshes and tries again, within the same deadline. - The wait, every retry and the start share one deadline. Each read is bounded by the time left, and the start receives only the remaining seconds. - As soon as any state other than `started` is observed, the lease's stored session is dropped before waiting, so a restart that happened outside the plugin never reuses a dead session. - Starting a settled sandbox (`startSettledSandbox`) is unchanged. - `plugin.test.ts`: - realize waits for a stopping sandbox, then starts it; - realize retries a start that was rejected for a state change; - realize gives up once the transition outlasts the deadline; - another start error fails at once; - the start receives only the budget the wait left; - a restart outside the plugin that the wait sees through opens a fresh session. - The tests stub the API key with `vi.stubEnv`. ## Verification - Red/green: on master, the wait and the retry cases fail with "Sandbox state change in progress". With the change they pass, and the deadline and other-error cases hold. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 614 tests passed (8 skipped). - `tsc --noEmit` for the Daytona plugin package: passed. ## Risks - Low risk. The wait is bounded by the start deadline the plugin already used. Only transitional states and the provider's state-change rejection are waited out, and every other error behaves as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:00:14 -06:00
DottaandClaude Opus 5.5	9c18cf61da	fix(runner): keep discarded OpenCode token deltas from delaying tool calls (#20209) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode runs use a TypeScript app-server proxy between OpenCode and runnerd > - runnerd reads the proxy's frames in order, only after the controller acknowledges earlier events > - For each frame, runnerd rewrites its whole durable provider state with fsync, including every tool result that the turn retains > - The proxy sent one `item/delta` frame for each reasoning or tool-output token, and runnerd discards `item/delta` > - A Paperclip tool call request therefore waited behind hundreds of discarded frames, and the wait grew with each tool result in the turn > - This pull request stops the proxy from sending the token deltas that runnerd does not consume > - The benefit is that tool calls reach runnerd without that queue, so agent turns finish faster ## Linked Issues or Issue Description **What happened** In the chat eval `plugins.configure-plugin` (runner-opencode, deepseek-v4.1-flash, GitHub Actions run 42574640038), `paperclip_search_api` calls in one turn slowed from about 1.4 s to 9-15 s each. The server handled every call in 50-250 ms (`tool.execution` spans). For one `call_api` call, 6.3 s passed between the server's 403 and the next agent activity. Time to first token was about 4.4 s. The run events show the cause. Events that OpenCode streams reach runnerd in bursts. In one burst, the usage events for three different model requests arrived within 0.5 s, which no model can do. Each burst ends with the next tool call request. 143 of 203 recorded runner-opencode runs that we checked show such bursts, and 108 show a gap of 8 s or more. Many chat eval cases that time out use this runner. The mechanism: 1. OpenCode streams reasoning one token at a time. The proxy forwarded each token as its own `item/delta` frame. 2. runnerd reads proxy frames in order and only after the controller acknowledges earlier events. For each frame it calls `save_state`. That serializes the whole provider state, with the tool results retained in the turn, and fsyncs it. 3. runnerd's Codex facade has no mapping for `item/delta`, so it discards the frame after the save. 4. A tool call request is one more frame in the same ordered stream. It waits until runnerd works through every earlier delta frame. The wait is about (reasoning tokens) x (save cost), and the save cost grows with each retained tool result. **Expected behavior** A Paperclip tool call reaches runnerd soon after OpenCode makes it. Its delay does not grow with the reasoning length or with the number of earlier tool calls in the turn. **Steps to reproduce** 1. Run a native OpenCode chat turn with a reasoning model that makes many sequential Paperclip tool calls, for example `search_api`. 2. Compare each tool's `item.completed` tool_result time with the `tool.execution.completed` time and with the next `item.started` tool_use time in the run events. 3. See that the gap grows during the turn, and that events for several model requests arrive within less than one second. 4. For a local model of the mechanism: run the pinned OpenCode binary with a controlled OpenAI-compatible provider that streams 300 reasoning tokens before each tool call. Pass the driver events through the proxy's mapping and batching into a reader that consumes one frame at a time with a save cost that grows by 4 ms per completed tool result. Before this change, 8 sequential calls waited 1.7, 3.1, 4.2, 5.4, 6.7, 7.9, 9.2 and 10.5 s behind 142-261 queued frames. After this change, they waited 64-93 ms behind 0 frames. ## What Changed - `shouldForwardOpenCodeProxyItem` now takes the event type and payload. For `item.delta`, it forwards only assistant text (`agentMessage` or `text`), which runnerd maps to `item/agentMessage/delta`. - The proxy keeps reasoning and tool-output token deltas. Completed reasoning and tool items still go to runnerd as `item/completed`. Tool progress still goes as `item/updated`. - New regression test: a step with 1,000 streamed reasoning and tool-output tokens puts the same number of frames before the tool call request as a step with 1 token, and no `item/delta` frame is written. - New test: assistant text deltas are still forwarded. Completed reasoning items are still forwarded. ## Verification - `pnpm exec vitest run src/cli/opencode-proxy-events.test.ts src/cli/opencode-proxy-output.test.ts` in `packages/paperclip-runner`: 15 passed. - Red/green: with the predicate change reverted, the two new tests fail. With the change, they pass. - `tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner` has no errors in the changed files. Local errors exist only in `src/eval/workflow-harness.ts`, which needs a built `@paperclipai/paperclip-eval-kernel`. - Local before/after measurement with the real pinned OpenCode binary (`opencode-ai@1.18.34`), described in Steps to reproduce, step 4. - Not run locally: the suites that need the `paperclip-runnerd` debug binary or the bundled provider entrypoints (`runnerd-opencode-streaming-tools`, `runnerd-codex-transport`, `opencode-proxy-thread-read`). They fail the same way on the unchanged base in this worktree because those build outputs are missing. CI covers them. ## Risks - Low risk. The changed frames were discarded by runnerd. No persisted state shape changes, so no previous-release story is needed. - If runnerd later maps a reasoning delta method, the proxy must forward it with batching, as it does for assistant text. The test names this boundary. - This change does not change the per-frame durable save in runnerd. Mapped frames (assistant text, tool items) still pay it. A runnerd change that skips the save when a notification changes no state would protect other proxies too. That is a separate change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, local benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 02:23:44 -06:00
DottaandClaude Opus 5.5	42bc238aa5	fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records it as `provider_turn_lost_on_restore` and continues in a same-conversation continuation turn > - #20118 taught the event consumer to skip a stale restart-lost terminal of another turn once a continuation is running. It applied only when the checkpoint already recorded the interruption > - When the checkpoint instead names the lost turn as active (the drain hit right after the first turn was submitted), the consumer starts the continuation on the first lost terminal. If the same lost terminal is then delivered again, it was taken as the run's terminal and failed the run > - This pull request applies the skip to any recovered run once its continuation has started. For a checkpointed interruption it still waits for the startup decision first > - The benefit is that a turn lost on restore always continues, however often its lost terminal is delivered ## Linked Issues or Issue Description Refs #20118 **What happened?** Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master 9b71d03215): 94 of 95 passed. In t90 (claude M1 parent), run `e5eab3cb`: 1. The run submitted its first turn at 06:28:25, and a graceful drain followed at 06:28:28. 2. After restore at 06:28:41, the ACPX runner replayed the turn's accept and start, then reported `turn_lab_1abbeca2…` as `provider_turn_lost_on_restore`. 3. The controller started continuation `turn_lab_a3d009c6…`; the saved checkpoint names it as the active turn. 4. The run failed at 06:28:45 with `native_provider_terminal_failed: The runner restored the conversation after process loss, but the previous turn is no longer active.` and needed a Retry. **Expected behavior** Once the continuation is running, a later delivery of the lost turn's terminal is history, not the run's terminal. The run finishes on the continuation. **Steps to reproduce** 1. Recover a run whose checkpoint names the interrupted turn as active (not yet recorded as terminal). 2. Deliver that turn's restart-lost terminal, which starts the continuation, then deliver it again before the continuation completes. 3. Before this change, the second delivery ends the run as failed. **Paperclip version or commit** 9b71d03215 (wave). Branch rebased on current master. ## What Changed - `native-session-runtime.ts`: `supersededByRestartContinuation` applies to every recovered run. - Once a continuation has started, a restart-lost `turn.failed` for any turn other than that continuation keeps the consumer going. - With a checkpointed interruption, it still waits for the startup decision first. Before this change it excluded the checkpointed interrupted turn itself, which could also end a run if that turn's replayed terminal arrived before the startup continuation began. - `native-session-runtime.test.ts`: - a lost terminal delivered again after the continuation starts; - the checkpointed interrupted turn's own terminal arriving before the continuation starts. ## Verification - Red/green: both new cases fail on master (the run ends failed or on the stale terminal) and pass with the change. The #20118 cases still pass. - `vitest run` over the runner restart, recovery, continuation and terminal test files: 9 files, 389 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm run typecheck:typescript` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The skip still requires a recovered run, a recoverable `provider_turn_lost_on_restore` failure, a continuation that actually started, and a terminal for a different turn than that continuation. A loss of the continuation itself, real provider failures, and runs that started no continuation keep their current handling. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 01:08:19 -06:00
Nicky LeachandClaude Opus 5.5	9046884811	perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner executes native agent runs. A host can park an idle Runner (release it and freeze it) between runs, and resume it for the next run on the same task. > - A follow-up run that reuses a parked Runner must claim it first. The claim is a bootstrap ticket, a checkpoint flush, and the plugin resume that thaws the Runner. > - Core made the claim only inside `attachRun`, after all of its per-run preparation. The claim does not depend on that preparation, so the two ran in series. > - This pull request starts the claim while Core still prepares the run. `attachRun` then waits for the claim that is already in progress. > - The benefit is a shorter start for every follow-up run on a parked Runner. The model below gives about 285 ms per run at P50. ## Linked Issues or Issue Description No public issue exists. The issue is described inline. **What happened?** A follow-up run that reuses a parked Runner claims the Runner only in `attachRun`. Core calls `attachRun` after backend setup: runner environment, assigned MCP tools, tool authority and the plugin task owner. The claim waits for all of this, but it uses none of it. **Expected behavior** The claim runs at the same time as backend setup. Everything that needs the new run's tool authority stays at attach. **Steps to reproduce** 1. Run a Codex or Claude task on a host that parks Runners between runs. 2. Let the run finish and the Runner park. 3. Send a follow-up. Before this change, the plugin resume starts only after backend setup ends. **Paperclip version or commit** `master` at 7086019d4b. **Deployment mode** Hosted, with parked Runner tasks. ## What Changed - **Transport** (`runnerd-codex-transport.ts`): - New `beginRunAttachment({ runId })`. It runs the attach admission checks, then starts the existing `#claimReleasedRunner` as the held attachment operation. It returns `false` and does nothing when the Runner is not released and confirmed, a claim already exists, or admission fails. - The admission checks move into `#assertRunAttachmentAdmission`. `beginRunAttachment` runs them first, and `attachRun` runs them again. - The next `attachRun` for the same run waits for the early claim. An `attachRun` for a different run, `release` and `turn/start` still fail with `native_runner_run_attachment_in_progress` while the claim is held. - After an early claim, `attachRun` fails with `native_runner_resume_reconciliation_expired` before it registers if the claim deadline has passed. - **Pass-throughs:** `HarnessNativeSession`, `CodexHarnessSession` and the session, harness and transport contracts get an optional `beginRunAttachment`. Each layer does the same identity, protocol-integrity and busy checks that its attach does. - **Core** (`native-session-executor.ts`): - The executor records when the in-memory warm entry it reuses was parked (`released`). - `createRunnerdBackend` gets `beginRetainedRunAttachment`. Core gives it only for that parked in-memory warm entry. - The plugin task owner preparation moves into `preparePluginTaskOwner`. On the early path, Core prepares and rebinds the retained owner at the same time as the runner environment and the assigned MCP tools, then starts the claim. - Cold, adopted, remote warm recovery and restart paths keep the old order. ### What stays at attach, and why - Registration, the quiescence probe, `run.attach` and the authority rotation stay in `attachRun`. Checkpoint registration needs the new run's tool authority epoch, and Core creates that epoch during backend setup. - The admission checks run again at attach, so a state change during preparation still blocks the attachment. ## Verification - `vitest run src/live/runnerd-codex-transport-release-resume.test.ts` (runner): 21 passed. Five new modes: - `early claim`: `attachRun` waits for the held claim and does not route the new run before the claim ends; - `early claim abandoned`: a close after the claim ends fails the waiting `attachRun` with `closed`, and the new run is never routed; - `early claim cancellation`: a close during the claim cancels it, and the next `attachRun` fails with `closed`; - `early claim expired`: the deadline passes during the claim, and attach fails before it registers; - `early claim stale ticket`: the deadline passes after the claim, before attach. - `vitest run plugin-runner-release-resume.integration.test.ts plugin-runner-eviction.integration.test.ts` (server, with real Runner processes from `cargo build --workspace --bins`): 2 files, 51 passed, none skipped. New modes: - `cached held claim` (Codex and Claude): the plugin resume is held, and the test proves Core's tool preparation runs while the claim is in progress; - `cached cancellation` (Codex): a cancellation after an early claim retires the Runner and the run stops cleanly. - `pnpm run typecheck:typescript` (runner) and `pnpm typecheck` (server): pass. - Evidence (modeled, not measured live): traces of 8 resumed runs on parked Runners in a hosted test deployment, on current `master`. The claim takes about 300 ms, and it fits inside backend setup in 7 of 8 runs. The modeled saving is about 285 ms per follow-up on a parked Runner at P50 (range 208–331 ms). I did not run this change on a hosted deployment. - Not run: the full server suite (CI runs it), smoke tests, and the hosted park, claim and attach loop with this change. ## Risks - **New failure mode:** Core now claims the parked Runner before backend setup ends. If that run then fails preparation, or is cancelled before attach, the existing retire path closes the claimed session. The Runner is retired, so that agent's next follow-up starts cold. Before this change, a preparation failure happened before the claim, while the Runner was still parked. - **Ordering:** the retained plugin task owner is rebound to the new run earlier, before Core creates tool authority. Nothing in the owner preparation reads that authority. A failure in tool authority setup now happens after the rebind, and the retire path handles it. - **Scope:** only an in-memory warm entry that was parked takes the early path. Cold launches, adopted Runners, remote warm recovery and restart recovery are unchanged. - **Deadline:** the claim deadline now starts earlier. A very slow backend setup can make the claim expire before attach. Attach then fails with `native_runner_resume_reconciliation_expired` before it registers, the same failure as a slow claim today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, git, GitHub CLI) and subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 22:56:08 -08:00
Nicky LeachandClaude Opus 5.5	92a18bb0f3	perf(runner): restart the ACPX provider when a turn settles (#20190) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner drives Claude through the ACPX sidecar. The sidecar's MCP endpoint lasts for the whole session, so it cannot tell which turn sent a late tool callback. > - Because of this, the Runner reaps the sidecar and provider of a generation that ran a turn before it releases call-ID tombstones and rotates receipt epochs. Today this restart runs at the start of the next turn. > - Every follow-up turn in a run pays that restart on its critical path: about 1.5 s with real Claude. #20145 moves the restart to park, but a second turn in the same run, after the first turn that follows an attach, still restarts cold. > - The restart does not have to wait for the next turn. When a turn settles and the host has its terminal result, the Runner is idle. > - This pull request restarts the provider at that point. The next turn, or the next park, then finds a fresh generation. > - The benefit: same-run follow-up turns start without a restart, park does no restart after a settled turn, and the late-callback guarantee is unchanged. ## Linked Issues or Issue Description Refs #20145, which moved the restart to park. This pull request builds on it. No public issue exists, so the problem is described here. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** `AcpxProviderSession::start_turn_timed` restarts the provider before any turn whose generation already ran a turn. A follow-up turn in the same run waits for `session.suspend`, sidecar shutdown, a new sidecar, `initialize`, `session.open` and `run.attach` before `turn.start`. Since #20145, park does the restart instead, but only for the first turn after an attach. **Proposed behavior** When a turn settles and the controller has ACKed every committed event, the Runner restarts the provider. The next `turn.start` uses the fresh generation and only rotates receipts. Park reuses the fresh generation and does no second restart. **Reason and benefit** The restart leaves the turn's critical path, both for same-run follow-ups and for the turns after a park. The order of reap and rotation stays the same, so a late callback from the reaped generation still cannot reach a later turn. ## What Changed - **Runner loop** (`durable/runner.rs`): new `CommandExecutor::prepare_next_turn` hook, a no-op by default. The runner calls it after the idle provider poll, only when no control frame is waiting and the controller has ACKed every committed event (`acked_source_seq >= highest_source_seq`). So the restart always runs after the terminal result reaches the host. - **ACPX turn-end restart** (`acpx_park_preparation.rs`, `acpx_provider_generation.rs`): - `restart_session_after_turn` restarts a Claude session that has a completed turn, nothing pending, no buffered events, a dormant goal and its whole event tail ACKed. It first settles an abandoned park observation. If that settlement reads more events, it skips this round, and the next idle round restarts after those events are ACKed. - `AcpxProviderSession::restart_after_turn` does the same suspend → reap → start → bootstrap → identity check as the turn-start restart. It records the reaped process id in `restarted_after_turn_from`. This field is the explicit "fresh because of a turn-end restart" marker. - The session does not poll a fresh generation until its next turn or rebind, the same as a prepared one. Its startup output is published with the next turn, as today. Polling it early would add events after the host computed its park cut. - **Next turn** (`acpx_provider_session.rs`): `start_turn_timed` restarts only when the generation ran a turn. When `restarted_after_turn_from` is set, it does the post-restart receipt and ledger rotation without a second restart. `session.reconciled` reports the reaped process as `previousProcessId`. - **Park**: `prepare_next_generation` restarts (with the 20 s park deadline) only if no turn-end restart ran. Otherwise it marks the generation for `run.attach` rebind and returns `false`, so the runner does not ask for a second park proof. The park-time receipt rotation moves to the next turn start, where the turn-end path does it too. - **Restart failure**: the executor sets the same state that a failed turn-start restart leaves (`closed`, no session, provider exit confirmed). The next `turn.start` fails with the same `ACPX turn start failed closed: …` error that the turn-start restart gives today. - **Fake sidecar**: new `pending-input` fault, which opens an input request in a turn that then completes. ### Races | Race | Handling | | --- | --- | | A turn or steer arrives during the turn-end restart | The runner loop is single-threaded. The command waits in the socket until the restart returns, then starts on the fresh generation without a restart. Only one generation is live. | | A turn arrives before the hook runs (for example, in the same read as the ACK) | Control frames go first, so the turn restarts at its start, as today. The hook then restarts after that turn. | | Park arrives during the turn-end restart | Host control is read at the top of the loop, so park waits. Park then reuses the fresh generation. No event is published in between, so the host's park cut still matches. | | Park arrives before the hook runs | Park restarts with the 20 s deadline, as in #20145. The hook then finds a fresh generation and does nothing. | | Eviction, `runner.suspend` or retire after the restart | They reap the fresh generation through the ordinary paths. | | The restart fails | The session is `closed`. The next turn fails with the same error and the same durable state as a failed turn-start restart. Park returns conflict, as it does for a closed session today. | | Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after the restart return `already_settled`, as after any settled turn. A cancellation that retires the runtime closes the session, so no restart runs. | | Interrupted, cancelled or failed turn | `completed_turn_observed` is false, so no turn-end restart runs. The next turn restarts at its start, as today. | | Pending permission or input requests at the terminal | The reducer ends them at the terminal, so the restart runs. A later `request.resolve` fails with `no active turn`, with or without the restart. | | Active session goal at the terminal | No restart, because the goal can keep working after its prompt. | ### Removed as redundant, and what stays - **Removed:** the park-time restart and dormant verify when a turn-end restart already made the generation fresh. The runner's `host_park_ready` proves dormancy just before `prepare_for_park`, and nothing runs in between. - **Removed:** the runner's second park proof in that case, because the provider did not change. - **Removed:** the park-time receipt and ledger rotation. The next turn start does it once, for both paths. - **Kept: the turn-start restart.** Without it, a turn queued before the hook, a turn after an interrupted or cancelled turn, a turn after an active goal, and any non-Claude ACPX turn would run in the generation that ran the previous turn. Tests cover the queued-turn and interrupted cases. - **Kept: the park-time restart.** Without it, a park that arrives before the hook would leave the next run to restart at attach. A test covers this. - **Kept: the settlement of an abandoned park observation.** Without it, the restart drops events that the old generation sent before the abandoned response. A test covers this, and it fails with the settlement removed. - **Kept: the goal-dormancy check.** Without it, the restart kills an active goal's autonomous work. A test covers this, and it fails with the check removed. - **Kept: the ACK gate.** Without it, the restart can run before the host has the terminal result. A runner test covers this. - **Kept: deferred polling of the fresh generation.** Without it, its startup event lands after the host computed its park cut, and the park conflicts. - **Kept: the kept failure message.** Without it, the next turn fails with `ACPX session is not open`, and the restart error is lost. ## Verification - `cargo test -p paperclip-runner-core --lib` and every `tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery` suite: all pass (debug). - New tests (`tests/support/acpx_turn_end_restart.rs`, run from `acpx_park_preparation`) prove these points: - A settled turn reaps the old pid before it spawns the new one, and no predecessor is still live. A same-run follow-up then sends only `turn.start` and reports the reaped pid in `session.reconciled`. - Park after a turn-end restart sends no sidecar command and returns `false`. Attach rebinds the same pid. The run's second turn restarts at its end, and the third turn starts without a restart. - The reaped generation's late callback never reaches the next turn. - A turn that arrives before the hook restarts at its start. A park before the hook restarts once. - A failed turn-end restart gives the same next-turn error, retry error, durable state and process count as a failed turn-start restart. - Cancellation after the restart is `already_settled`. `runner.suspend` reaps the fresh generation. - An active goal skips the restart. An input request open at the terminal behaves the same with or without the restart. - `acpx_provider_turns`: a session-level turn-end restart drops the old generation's late `issues.delete` callback, which reuses the previous turn's call ID. The next turn then admits that call ID from the fresh provider. Interrupted and retired turns do not restart at turn end. - `acpx_park_readiness`: a turn-end-restarted generation is park-ready. An abandoned park observation tail is delivered before the restart. - `durable::runner`: the hook runs only after the terminal is ACKed and when control is idle. It never runs during ACK debt, a waiting control frame or a disconnect. - Server integration tests (`plugin-runner-release-resume`, `plugin-runner-eviction` and `plugin-runner-task-warm`) with this branch's Runner: 49 passed. The provider process counts do not change: each released run still adds one process start, now at turn end instead of at park. - `cargo fmt --check`: clean. - Fake-sidecar timing (release build, 7 runs, medians): | | before (`master` with #20145) | this PR | | --- | --- | --- | | same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms | | second turn after an attach, `turn.start` → `turn.started` | 270.3 ms | 23.5 ms | | turn-end restart, after the terminal is ACKed | — | 29.0 ms | | park preparation after a settled turn | 34.1 ms | 0.03 ms | - Real Claude: not run for this pull request. With real Claude, the restart took about 1.5 s on the turn-start path, and this change takes it off that path. That number is not measured on this branch. ## Risks - A run that ends after one turn without a park now starts one extra sidecar that no turn uses. The Claude CLI itself starts only at the first prompt, so the cost is the sidecar's start and bootstrap. - The restart blocks the Runner loop for its length (about 1.5 s with real Claude), as the turn-start restart does today. It has no deadline, as before. A park that arrives during it waits, inside the host's 30 s park wait. - The fresh generation is not polled until its next turn. If it exits while idle, the next turn finds out at `turn.start` and fails closed. Park finds out at its observation. - The turn-end restart is for Claude only, the same as park preparation. Other ACPX agents keep restarting at turn start. - If the Runner crashes during the restart, the durable state still says `session_open`. Recovery is the same as for a crash while idle. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 22:34:26 -08:00
Nicky LeachandClaude Opus 5.5	2d93413085	perf(runner): prepare the next Claude provider generation at park (#20145) ... ## Thinking Path > - Paperclip runs agents through Paperclip Runner, and a host can park an idle Runner between runs and resume it for the next run on the same task. > - For Claude (ACPX), a follow-up run on a parked Runner pays about 1.5–2.3 s at `run.attach` before the provider can start the turn. The attachment closes the Claude sidecar and spawns a new one: initialize, `session.open` with `session/load`, `run.attach`, then a dormant verify. > - The respawn is required. The tool-bridge MCP endpoint lasts for the whole session and can't tell which turn a late callback came from, so a new turn must not run in a process that ran an earlier one. > - Nothing requires the respawn to happen at the *start* of the next run. A turn that ends 2 s later is a better experience than one that starts 2 s later. > - This pull request does the respawn while parking. The next run's `run.attach` then only rebinds the run id on a fresh, never-used generation. > - Follow-up Claude runs on a parked Runner reach the provider about 1.5–2 s sooner, and the late-callback guarantees are unchanged. ## Linked Issues or Issue Description No public issue exists. The issue is described inline. ### What happened? On a hosted deployment with parked Runners, Claude follow-up runs reached the provider about 1.8 s later than Codex follow-ups. `runner.session.resume` medians were 5.2 s for Claude and 3.4 s for Codex. The extra time is the ACPX respawn inside `run.attach`. Codex keeps its process in place. ### Expected behavior The respawn happens while the Runner parks, so the next run starts on a prepared generation. ### Steps to reproduce 1. Run a Claude (ACPX) task on a host that parks Runners between runs. 2. Let the run finish and the Runner park. 3. Send a follow-up. `run.attach` closes the provider and starts, initializes, opens and attaches a new sidecar before `turn.start`. The Claude CLI itself starts at the first prompt on either path. ### Paperclip version or commit `master` at e194e193. ### Deployment mode Hosted, with parked Runner tasks. ## What Changed - **`durable/runner.rs`:** - New `CommandExecutor::prepare_for_park(deadline)` hook, a no-op by default. The runner passes a deadline 20 s after the park request; the host waits 30 s for the reply. - The Park host-control arm calls it after `host_park_ready()`, re-proves readiness, and only then replies `parked`. - A failure records a diagnostic and replies `conflict`, the existing park-failure path. - **ACPX, park side** (`acpx_park_preparation.rs`, `acpx_provider_generation.rs`): - For Claude, with the session open, nothing pending and nothing queued: reap the old generation, using the same suspend-and-reap sequence a second turn uses. - Start the replacement with the current run attached, and prove it dormant. - Roll the tool receipt epochs and the turn-identity ledger over. - Record a digest of the session configuration it was opened with, ignoring the MCP binding id and the aggregate digest. - Every sidecar request during preparation is capped at the park deadline. Running out of time fails like any other preparation failure. - **ACPX, attach side:** - A prepared, never-used generation whose digest matches the next run is rebound: the sidecar gets `run.attach` with the new run id, the reducer and tool bridges are reset, and the snapshot is checked for the new run id. - Anything else takes today's cold path: a different tool set, skill, prompt or launch; a buffered callback; or any rebind error. - **ACPX, turn start:** the provider restarts when its generation has run a turn, instead of when the run has settled turns. `has_settled_turns` is removed. - **ACPX, polling:** polling skips a prepared generation until it is rebound or starts a turn, so nothing it emits is attributed to the parked run. - **Sidecar** (`acpx-runtime-sidecar.ts`): rejects `run.attach` once a turn has started in that process. - **Fake sidecar:** reports live predecessors at process start, and adds the faults `reject-open`, `late-tool-after-suspend` and `pre-turn-tool`. ## Verification - `cargo test -p paperclip-runner-core`: 1029 passed (debug). - New `tests/acpx_park_preparation.rs` (10 tests). These prove: - park reaps the old pid before spawning the new one, and acks only after the dormant verify; - a late callback from the old generation after park is rejected, and so is a pre-turn call from the new generation; - a compatible attach keeps the pid, and the sidecar and snapshot carry the new run id; - changed instructions, tools, skills or runtime context take the cold path; - a used generation is never rebound; - a prepare failure produces a park conflict and a recoverable state; - Retire works after a prepared park; - a replacement whose `session.open` stalls fails at the park deadline, leaves the session suspended, and the next run attaches and completes a turn. - `cargo fmt --check` and `cargo check --workspace --locked`: clean. - `vitest` `acpx-runtime-sidecar.test.ts` and `acpx-sidecar-input.test.ts`: 126 passed. The new re-attach guard test fails if the guard is removed. - `vitest` `plugin-runner-release-resume.integration.test.ts` and `plugin-runner-eviction.integration.test.ts` with the branch's Runner: 48 passed. Both now expect the provider process to be replaced at park instead of at attach. - Fake-sidecar timing (release build, 3 runs): `run.attach` took 230–260 ms cold and 40–80 ms after a prepared park. - Real Claude (claude-agent-acp 0.73.0, release build on a hosted test VM, 3 runs each way plus one with a 180 s cgroup freeze between park and attach): | | before | this PR | |---|---|---| | `run.attach` | 1691 ms mean | 212 ms mean | | `run.attach` start → follow-up `turn.started` | 2565 ms mean | 1065 ms mean | | `turn.start` → first provider output | 2261 ms mean | 2232 ms mean | | park preparation added to park | — | about 1450 ms | Session recall, an authorized tool call after the rebind, and the absence of any tool call or event from the old run all passed in every run, including after the freeze. The prepared generation holds only the sidecar: the Claude CLI starts at the follow-up prompt on both paths, so a parked Runner also holds one node process instead of the whole Claude tree. - Hit rate: on a hosted test deployment, all 15 consecutive Claude runs across two tasks carried identical content-addressed instruction and skill bundle paths, so the prepared digest would have matched. - Not run: the hosted park, claim and attach loop with real Claude, a native MCP binding or skills with real Claude, `cargo test --release`, smoke tests. - Pre-existing flake: `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` fails intermittently on clean `master` when the `hosted acpx durable::` subset runs in parallel. It passed in both full runs. ## Risks - **Park takes about 1.5 s longer** before it replies with real Claude. Preparation is capped at 20 s; the host's park wait is 30 s. - **Configuration changes:** a change between park and the next run (registered asset paths, tools, prompt) takes the cold path. That's safe, but saves nothing. - **Stale environment:** a prepared process was opened with the previous run's environment and disk state. If credential or configuration files change on disk while the Runner is parked, the prepared process won't see them, whereas the cold path would. - **Failed prepare:** a park that fails after the reap leaves no live provider. The next turn or `run.attach` starts Claude cold, as today. - **Scope:** only ACPX Claude prepares. Other providers park exactly as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, git, GitHub CLI) and subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 21:38:57 -08:00
DottaandClaude Opus 5.5	5a51a7685d	fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop cancels the scope's running work; the board resolves it with Raise budget and resume > - In the restart-chaos H1 scenario, the cancelled tasks did not continue after resume and each needed a manual Retry > - Resume is the board's explicit decision to continue, so the work it stopped should continue once, as if the board clicked Retry > - This pull request files that Retry automatically, keyed by the cancelled run, once the run's stop has settled > - The benefit is that a budget raise restores the work without manual Retries, without repeating completed work, and without charging the run's attempt budget ## Linked Issues or Issue Description Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101 (superseded: split into #20143 and this PR) **What happened?** Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802 (H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run (enforcement stop) and CHA-1's run (agent lifecycle pause from the budget hold) stayed cancelled until someone clicked Retry. **Expected behavior** Resuming continues each task its budget stopped, exactly once, with no attempt charged, an activity entry, and no repeat of work that already continued. A run a person stopped stays stopped. **Steps to reproduce** Set a budget below spend while a task runs; let the stop cancel the run; resolve with Raise budget and resume. Before this change, no new run is queued. ## What Changed - New table `budget_resume_claims` (migration 0451). One row per cancelled run, outside the run's result JSON, so a Stop that settles during the resume cannot erase it. - `markBudgetCancelledWorkForResume` (called by `resolveIncident` before the incident closes): claims every task run its budget stopped since the incident opened. That covers runs budget enforcement fenced, including stops still in flight, and runs the agent lifecycle pause stopped for this budget (`pauseStop`, carried from the lifecycle operation's `pauseCause` so a resume before the Stop keeps the link). Runs a person stopped are excluded. Two indexed passes led by `(company_id, status)`. - `deliverBudgetResumeContinuations`: once a claimed run's stop settles, files the resolving board member's Retry (`requestTaskRetry`, idempotency key `budget-resume:<run>`, with the member's session source and the chat Retry callback). The Retry path checks, under the task lock, that the run is still the task's latest and the task is open, so continued or completed work is never repeated. Claim settlement and its activity entry commit together; retryable refusals and errors retry. - Delivery runs in the always-on budget-enforcement worker after each pass's Stop delivery. Its schedule includes deliverable claims and the earliest stale-claim expiry. The execution-continuation worker is a backstop. - `agent.paused` / `agent.resumed` logging is in #20113. `budget.cancelled_work_resume_requested` (incident) and `budget.cancelled_work_continued` / `_not_continued` (task) are recorded here. ## Verification - New `budget-resume-continuation.test.ts` (Postgres, real `resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry` into a queued run): - each task continues once, including a blocked task; repeat delivery or resolve saves nothing new; - a stop still in flight at resume, whose terminal write replaces the result JSON, is still continued; - a 30-task scope is fully continued; - a lifecycle-pause stop caused by the budget is continued, and one caused by a manual pause is not; - continued work, done tasks, a run that finished before its stop, and person-stopped runs (including ones enforcement also fenced) are not continued. - `lifecycle.test.ts`: the pause records its cause on the operation and on stopped runs, including when the budget is resumed before the pause's Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery, and a wake at claim expiry. - 287 tests across the affected suites passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (Phase 3 runs it after merge). ## Risks - Resume now continues budget-stopped tasks automatically. A board that wants a task stopped can Stop it again; person-stopped runs are never continued. - A new table and one index (migration 0451). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 22:56:28 -06:00
DottaandClaude Opus 5.5	b54317d3fb	fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Paperclip Runner (native) run prepares its workspace and environment before it starts its first provider turn. This can take about a minute. > - A user's message that arrives during that time waits as a deferred wake behind the run. The Interrupt control then cancelled the run before any provider turn. > - The cancel threw away the whole preparation. A new run had to prepare again, so the user waited longer for the first token. > - This pull request gives the run's first turn the queued messages. The run seals its first-turn input immediately before `turn.start`, so the messages are part of the prompt that the turn starts with. > - The benefit is that the user sees the message handled in the same run, with no cancel and no second preparation. ## Linked Issues or Issue Description **What happened?** A Paperclip Runner run took about 54 s to prepare (environment startup, workspace, runtime selection). The user sent a second message 32 s into that time and clicked Interrupt 21 s later. Interrupt cancelled the run as `operator_interrupted` ("Interrupted to send queued messages") before any provider turn. A new run then prepared again from the start. The runtime-selection transaction that froze the run's input completed about 5 s before the click, and the coordinator claim came 4 s after it, so the message could not be given to the first turn at any point in the old code. **Expected behavior** When a user message arrives for a task whose current run has not started its first provider turn, the run does not stop. The message goes into that run's first turn. A run that is in a turn is interrupted as before. **Steps to reproduce** 1. Assign a task to a Paperclip Runner agent with a sandbox environment, so preparation takes a long time. 2. While the run prepares, post a second comment on the task. 3. Click Interrupt on the queued message before the run starts its provider turn. 4. See that the run is cancelled and a new run prepares again. **Paperclip version or commit** Staging revision b43cccbb0. ## What Changed - `server/src/services/native-runtime/first-turn-queued-input.ts` (new): `claimFirstTurnQueuedInput` seals a run's first-turn input in one transaction. It locks the task, the deferred wakes, and the run in the same order as the queue routes. It takes the responsible user's plain queued comments in queue order (not interaction answers, interrupt intents, independent continuations, conversations, or other users' messages). It renders them with source-trust sanitizing and secret redaction, then writes a `firstTurnQueuedInput` receipt (`paperclip.first-turn-queued-input.v1`) into the run context. A second call for the same run returns the same messages. `runAcceptsFirstTurnQueuedInput` is true for a running native (or preparing native) run that has no receipt and no accepted provider turn. - `packages/paperclip-runner/src/native-session-runtime.ts`: new `getFirstTurnQueuedInput` option. The runner calls it immediately before the first `session.startTurn` and appends the result to `task.prompt` (or to `events` for a resumed-session continuation envelope). It is not called for the disposition-only recovery turn. - `native-session-executor.ts`: passes the claim to the runner. A failed seal commits nothing, so the executor logs it and starts the turn without the messages. The messages stay queued. - `queued-comment-delivery.ts`: a message in a run's receipt counts as delivered only after the provider accepted that run's turn (`turn.accepted` or the provider's `turn.started`), and only if the message was not edited after the seal. Release promotion, Send, and later first turns use this rule. `explicit-native-continuation.ts` uses the same rule for stopped native runs. - `routes/issues.ts` Interrupt: when the target run has not sealed its first turn and has no accepted turn, the route does not record interrupt intent and does not cancel the run. The response has `executionWait.reason = "first_turn_pending"`. Other runs are interrupted as before. - Queue views (`routes/issues.ts` and the wake-queue adapter) hide messages that the active run took into its first turn. Queue edits keep those ids on the wake, so a run that fails before acceptance still leaves them queued. - `run-state.ts`: the issue-control run projection includes the receipt, so the queue poll can hide taken messages. - `doc/execution-semantics.md`: new section "Queued messages before a native run's first turn". ## Verification - Red before the fix, green after: - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/issue-queued-comments-routes.test.ts -t "first turn|already started"`: Interrupt on a preparing run and on a claimed native run before `turn.start` does not stop the run, and the first turn gets both messages in order. Before the fix, both cases cancelled the run. - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts -t "first-turn queued input"`: the claim runs immediately before `turn.start`, and the first turn's prompt ends with the queued messages. Before the fix, the claim was never called. - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/first-turn-queued-input.test.ts`: a run that fails before `turn.start` leaves the message queued. The next run delivers it, and after that no later run or copied retry context takes it again. Before the delivery rule, the "exactly once" and "accepted then failed" cases failed. - Safety cases that pass before and after: Interrupt still cancels a run whose provider accepted a turn (including a run saved by the previous release with no receipt) and a run whose first-turn input is already sealed. The existing "explicit interrupt fallback" cases for preparing and native runs without a responsible user still pass. - Speed: the seal is one short transaction before `turn.start`. In the embedded-Postgres test it took 5.5 ms. It removes a full second preparation (54 s on staging) from the message-to-first-token time. - Related suites: `src/modules/wake-queue`, `explicit-native-continuation`, `issue-queued-comment-queue`, `issue-queued-comments-routes`, `queued-comment-consumption`, `heartbeat/run-state` (888 tests), `native-session-executor.test.ts` (1306 tests), and the full `native-session-runtime.test.ts` (238 tests) pass. `pnpm --filter @paperclipai/server typecheck` passes. - Not done: no live run against a real provider or sandbox. Staging was read only. ## Risks - Previous-release state: a deferred wake saved by the previous release has the same shape and is taken normally. A run saved by the previous release has no receipt. If it is resumed before its first turn, it seals on the new release. If its provider already accepted a turn, Interrupt stops it as before. A run saved by the previous release while it was in `turn.start` but before acceptance counts as unsealed, so in that short window an Interrupt does not stop it and the message is delivered at the turn boundary. - Rollback: if the server rolls back after a run sealed and its turn was accepted, the old release does not read the receipt and can deliver those messages a second time. - A first turn that does not use the fresh-turn path (restart continuation, disposition-only recovery) does not call the claim. Its messages are delivered at the turn boundary as before. - A message that arrives after the seal waits for the turn boundary. Interrupt stops the run as before. - No schema migration. The receipt is a key in the run context. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution, read-only browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 22:44:18 -06:00
DottaandClaude Opus 5.5	28e070f8dc	feat(skills): show a line diff for a pending tracked-source update (#20155) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Skills can come from a tracked GitHub source, and an update moves the installed copy to a newer upstream revision > - Before an update, a person asks the agent "what changed in the new version?" > - The product has no diff for a pending update. The agent previews two full SKILL.md snapshots and compares them by eye > - In two eval runs the agent reversed the same changed line. In a third run it fetched the files with curl and ran `diff` itself, and the turn took too long > - This pull request adds a read-only endpoint that returns the line diff between the installed and the latest tracked revision > - The benefit is that review answers come from an exact diff, and they come faster ## Linked Issues or Issue Description No public issue exists. Feature description: **What happened** An agent asked to show what changed in a pending skill update had only `POST /skill-sources/preview`, which returns one file at one commit. It previewed both commits and compared about 9 KB of text by eye. It reported one changed line in the wrong direction ("every new build" -> "every build or release" instead of the reverse), and later retracted a real change. **Expected behavior** The agent can request the exact changes of a pending tracked-source update and report them with the correct direction. **Steps to reproduce** 1. Install a skill from a GitHub source at an old commit and set the tracking reference to a newer commit (automatic updates off). 2. Ask the chat agent: "Show me what changed in the new accessibility skill version before we update it." 3. See that the agent can only preview full files at each commit and must compare them itself. ## What Changed - Add `GET /api/companies/{companyId}/skills/{skillId}/update-diff`. For a skill from a tracked GitHub source, it compares the installed source revision (`currentRef`) with the latest tracked revision (`latestRef`). For each changed package file it returns `change` (added, removed, modified, or `not_compared`), `binary`, `executableChanged`, and a unified line diff (`-` installed, `+` latest), or `diffUnavailable` (`binary`, `too_large`, `omitted`, `budget`) that says why there is no diff. The endpoint is read-only. The `skillId` parameter handler checks company access and skill visibility, the same checks other skill reads use. - `hasUpdate` is false and `files` is empty when the installed revision is already the latest one. That case reads only the latest commit and scans nothing. - Return `supported: false` with a reason for: a skill without a tracked source, a disconnected source, an unknown installed revision, and a repository package that a scan could include only in part (a subset, or skill folders only). In the last case, files that a scan limit left out would otherwise look removed. - A path the scan left out of a revision (too large, a symlink, or rejected by the audit) is listed as `not_compared` instead of added or removed. - One response shares a work budget (24M LCS cells), a diff-text budget (1M characters) and a file-count budget (1,000 files), and sets `truncated` when it reaches one. The GitHub read and both scans stop when the caller disconnects. - Add `server/src/services/skill-update-diff.ts` with a bounded line diff (shared head and tail trimmed, LCS size capped). An absent side has no lines, a final newline is not a line, and a missing final newline shows the unified-diff marker. A replaced line reads `-old` then `+new`. - Add the shared types `SkillSourceUpdateDiff` and `SkillSourceUpdateDiffFile`. - Document the endpoint in the API catalog (`openapi.ts`) so agents find it with `search_api`. ## Verification - `cd server && npx vitest run src/__tests__/skill-update-diff.test.ts` (10 tests): - direction of `-`/`+` lines, and separate hunks with bounded context - exact diffs for added and removed files, and final-newline handling - a change to only the executable bit, and paths left out of a scan - the response budgets - the snapshot comparison for the selected skill path, with the abort signal passed to both scans - a commit mismatch - refusal of partial repository scans - `cd server && npx vitest run src/__tests__/skill-update-diff-routes.test.ts` (6 tests): - the diff response - no pending update (no scan) - a caller from another company (board user and agent) gets 403 before any lookup - a skill the caller cannot read gets 404 before any source read - every unsupported case - a client disconnect aborts the scans - `company-skills-routes.test.ts` and `openapi-routes.test.ts` pass (119 tests across the four files). - `npx tsc --noEmit -p server`: no errors from this change. - End to end: the `skills.review-update` chat eval exercises this path. Its grader checks the quoted hunks against the real pinned diff. ## Risks - Low risk. The endpoint is new and read-only. Existing routes do not change. - When an update is pending, each call scans the source twice (installed and latest commit). It uses the same GitHub reader, scan limits and scan slots as preview and refresh. The diff work and response size are capped per response. - `/update-status` and this route both read the latest commit of the tracking ref. They are kept separate because `/update-status` compares with the source's last scan, while this route compares with the installed revision. - No UI uses the endpoint yet. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 21:55:52 -06:00
DottaandClaude Opus 5.5	cedf335c98	fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records that turn as failed with `provider_turn_lost_on_restore` and continues on a same-conversation continuation turn > - Restart recovery blocks a run as `provider_checkpoint_permanently_failed` when its checkpoint carries a failed run terminal and no pending restart interruption > - While a continuation is running, the checkpoint keeps the lost turn's failed run terminal and names the continuation as the active turn. The pending-interruption check returns nothing once a turn is active > - So a second restart during the continuation read the run as permanently failed, blocked replay, and needed the board > - This pull request treats that failed terminal as the restart loss it records whenever an active turn follows a restart-lost terminal turn > - The benefit is that runs survive repeated restarts during their continuations, including M1 parents and their subtasks ## Linked Issues or Issue Description Refs #20148 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master 8e3b64a257), t05 (claude M1 parent): 1. Run `e1e37dab` lost turns to restarts at 00:41:32 and 00:47:53. Each time it continued, and the third continuation `turn_lab_92b03dc7…` was active. 2. At 00:48:29 the next restart blocked it: `native_restart_recovery_blocked` / `provider_checkpoint_permanently_failed`, with replay blocked. 3. Its saved checkpoint held `terminal.runTerminalState: "failed"` (from the lost turn), `activeTurnId` set to the continuation, and a last terminal turn fingerprinted `provider_turn_lost_on_restore`. Its codex/ACPX subtask's run `f9b10d06` was blocked in the same pass with the same shape, and its continuation had already proposed a semantic result. Both stayed blocked. The parent's Retry was held on the blocked child. **Expected behavior** A failed run terminal that records a restart loss, followed by an active continuation turn, is not a permanent failure. Recovery claims `resume_dead_runner`. The runtime then reconciles the active turn, and a held result settles a lost turn as completed. A real provider failure followed by a later turn still blocks. **Steps to reproduce** 1. Checkpoint a run whose last terminal turn is a restart loss, whose `terminal` is failed, and whose `activeTurnId` is a later continuation (optionally with a semantic result). 2. Claim restart recovery. Before this change it blocks with `provider_checkpoint_permanently_failed`. **Paperclip version or commit** 8e3b64a257 (wave). Branch rebased on current master. ## What Changed - `contracts/native-session-backend.ts` (runner): the new `nativeRestartContinuationInFlight` is true when all of these hold: - the checkpoint has an active turn; - its last terminal turn is a different turn; - that turn's fingerprint is a failed, result-less, recoverable `provider_turn_lost_on_restore`. - `native-session-runtime.ts`: recovery clears that stale failed terminal for a continuation in flight, as it already does for a checkpointed interruption. Before, it threw `provider_checkpoint_failed_terminal` before it reconciled the active turn. - `native-restart-recovery.ts` (server): `checkpointFailed` now also requires that no restart continuation be in flight. It uses the same contract function through the vendor boundary. - `native-session-runtime.test.ts`: an end-to-end recovery test. A checkpoint with the stale failed terminal and an active continuation resolves on the continuation's result without starting a new turn. - `native-runner-restart-recovery.integration.test.ts`: - a codex case of the stale-terminal shape as saved in production; - an ACPX case with a held result; - a held case where an active turn follows a real provider failure (`model_not_found`). ## Verification - Red/green: on master, the stale-terminal case is blocked with `provider_checkpoint_permanently_failed`. With the change it claims `resume_dead_runner`. The real-failure case stays blocked on both. - Red/green (runtime): without the runtime change, the end-to-end test throws `provider_checkpoint_failed_terminal`. With it, the run completes. - `vitest run` over the restart-recovery and failed-workspace-recovery test files in `server/src/services/native-runtime`: 5 files, 305 tests passed (14 skipped). - `vitest run` over the runner restart, recovery, continuation and terminal test files: 9 files, 385 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`) and `pnpm run typecheck:typescript` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The exception applies only when the recorded failure is itself a recoverable restart loss and a later turn is in flight. Every other recovery gate is unchanged: the provider-attempt budget, controller takeover and process proofs. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 20:28:12 -06:00
Devin FoleyandPaperclip	546cfee919	fix(github): send gh repo create and fork to the connection's tools (#20142) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents with a managed GitHub connection get `git` and `gh` launchers that use the connection's credential > - The launcher gives that credential to any `gh` command, and the broker does not see which command runs > - So `gh repo create` and `gh repo fork` skip the connection's tool switches, ask-first approvals, owner defaults, and audit > - The connection's `create_repository` and `fork_repository` tools apply all of these > - This pull request stops `gh repo create`, `gh repo new`, and `gh repo fork` when a managed credential is in use, and names the tool to use instead > - The benefit is that repository creation follows the connection's rules, whichever path the agent tries first ## Linked Issues or Issue Description Refs #20141 (default owner for `create_repository` and `fork_repository`). **What happened?** An agent with a managed GitHub connection can run `gh repo create` in its shell. The command uses the connection's token, but it does not use the connection's tool rules. If the user turns off or reviews `create_repository` on the connection, the agent can still create a repository with `gh`. The owner default from #20141 also does not apply, so the command can fail with `403 Resource not accessible by integration` when the App is installed only on an organization. **Expected behavior** With a managed connection, repository creation and forks go through the connection's tools. The tool switch, approvals, owner default, and audit apply. **Steps to reproduce** 1. Connect GitHub with a managed connection. 2. Set the connection's `create_repository` tool to off or ask first. 3. In an agent run, run `gh repo create demo --private`. 4. The command runs with the connection's credential and ignores the tool setting. ## What Changed - `packages/adapter-utils/src/github-launcher.ts`: when the broker returns a managed credential, `gh repo create`, `gh repo new`, and `gh repo fork` exit with code 1 before `gh` starts. The message names the `create_repository` or `fork_repository` tool and tells the agent to finish with Git. - `--help` still runs. Other `gh` commands do not change. Executions without a managed credential run these commands as before. - `doc/execution-github-identity.md`: describes the behavior and states that this is guidance, not a security boundary. ## Verification - New test in `packages/adapter-utils/src/github-launcher.test.ts`: "sends repository creation and forks to the connection's tools when a connection is in use". It covers `create`, the `new` alias, `fork`, `--help`, another `gh repo` command, and the case without a connection. It fails without the launcher change. - `npx vitest run src/execution-target.test.ts src/github-launcher-environment.test.ts src/github-launcher.test.ts` in `packages/adapter-utils`: 83 passed. - `npx vitest run src/drivers/acpx/github-launcher.test.ts` in `packages/paperclip-runner`: 3 passed. - `npx vitest run src/services/heartbeat-github-launchers.test.ts` in `server`: 12 passed. - `npx tsc --noEmit -p packages/adapter-utils` passes. ## Risks - Agents that used `gh repo create --source . --push` must now call `create_repository`, then add the remote and push with Git. The message tells them to do this. - If the connection's tool is off for the agent, the agent cannot create a repository. This is the purpose of the tool setting. - This is not a security boundary. Other commands that receive the token, such as `gh api`, can still call the same GitHub endpoints. - Running launchers keep their earlier source until the next run stages new launchers. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-08 17:08:35 -08:00
DottaandClaude Opus 5.5	c49ab6b9ee	feat(issues): wake the parent once when a child or blocker stalls on the board (#20098) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task delegates to children and waits on blockers. Its agent learns of progress only through wakes > - Today only completion (`issue_children_completed`) and resolution (`issue_blockers_resolved`) wake a waiting issue > - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on a recovery action that needs the board, and its parent sat silently blocked for 50+ minutes > - This pull request wakes the waiting agent once per such stall, tells it what it can do, and shows the stall on the parent's page with the child's Retry > - The benefit is that a stuck child or blocker no longer silently strands its parent ## Linked Issues or Issue Description Refs #19831 (durable ownership plan; restart chaos campaign). Restart chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude). **Problem** From the kept database: - CHA-8 `blocks` CHA-5. - CHA-8's board-owned recovery action (`native_session_retry_exhausted`) was created at 18:44:02.909 and resolved with outcome `blocked` at 18:44:08, when CHA-8 moved to `blocked` for the board. - CHA-5's only wake in the whole wave was its `issue_assigned` at 18:20:48. Its agent never heard about CHA-8. **Expected behavior** - When a child or blocker stalls on the board, the waiting agent is woken once. It can then re-delegate, escalate, or block with a reason. - The board sees the stall on the waiting task, with the child's Retry. ## What Changed - **Shared:** - `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`), `ISSUE_BLOCKER_STALL_RELATIONS`, and the types `IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`; - `Issue.stalledDependencies`. - **Server, `services/recovery/blocker-stall-wakeups.ts`:** - A stall is an issue that is not done or cancelled, whose latest recovery action needs the board. That means either open (`active`/`escalated`) and board-owned or escalated, or board-owned and resolved as `blocked` while the issue is still `blocked`. The second case is what t05 needed: recovery resolves its action that way when it hands the issue to the board. - Waiters are the parent, and every issue the stalled one `blocks`, when they are open, visible and agent-assigned. - Each waiter gets one wake per stall. The idempotency key `issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked through the indexed `(company_id, payload->>'issueId')` path. A later stall has a new action id, so it wakes again. - Each wake writes activity `issue.blocker_stalled_wake`. - The pass runs from the recovery-action sweep (`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep (`reconcileResolvedDependencyWakeBackstop`, which visits each blocked issue). No new worker. - A waiter's agent that cannot read the stalled issue (`issue:read`, which honors private visibility) is woken without its title, identifier or recovery details; the activity entry withholds them too. - **Database:** migration `0450` adds the partial unique index `agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id, idempotency_key)` for `issue_blocker_stalled:` keys in delivered or pending statuses, so two concurrent sweeps cannot both deliver a stall. It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`). - **Queue:** `allowsDependencyBlockedWake` (in `run-dispatch/domain/wake-context.ts`) lets a stall wake run while the waiter's blockers are unresolved, at both dependency gates. Without it the wake was parked until the stalled blocker resolved, which is never. Pause holds are unchanged. - **Agent guidance:** the wake payload carries `blockerStall`, and `renderPaperclipWakePrompt` adds a "Stalled dependency" section. It names the task and its recovery action and the board's next action, then the three choices: re-delegate, escalate with a comment, or block with a reason naming the stalled task. - **Other reason lists:** `issue_blocker_stalled` joins the responsible-user wake reasons, the conversation execution-wake reasons (it never drives a chat turn) and the wake diagnostic's known reasons. - **Board surface:** - `GET /issues/:id` returns `stalledDependencies`, readable issues only (`readableIssueEdgeIds`). - `StalledDependencyNotice` shows on the issue's chat notices, next to `IssueBlockedNotice`: each stalled child or blocker, its board next action, a link, and the child's own Retry (`useTaskRetry`), offered only when the child's interruption allows it. ## Verification Red/green: - `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests). - On master: no `issue_blocker_stalled` wake is raised. - With this PR: the parent is woken once per stall, and re-woken after a new stall. A blocker's dependent is woken. - The t05 shape (an action resolved as `blocked`) wakes once, and nothing more after a Retry. - No wake for an agent-owned action, a done, unassigned or hidden waiter, or an action resolved another way. - `listStalledDependencies` lists children and blockers. - `heartbeat-dependency-scheduling.test.ts` "admits an issue_blocker_stalled wake while a blocker is unresolved". On master the wake was parked (`null`); once admitted, the run-start gate cancelled it. With this PR it runs with `dependencyBlockedInteraction`. The file: 14 passed. - `packages/adapter-utils` `server-utils.test.ts`: the new prompt test failed on master (no `blockerStall`). The file: 141 passed. - `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed. `IssueChatThread.test.tsx`: 109 passed. Regression and static checks: - Nearby server suites (recovery durable wait, dependency wake routes, recovery actions, run-preparation): 149 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. - `pnpm check:token-gates` is clean. Review round 1 (Greptile, Superagent), each with a regression: - A backlog waiter is never woken. The waiter's status and assignee are guarded at enqueue. - One stall per run: no coalescing, so a merged run cannot keep only the last stall's context. - Only a delivered or pending wake ends a stall. A refused one (wake-on-demand off) is retried after 10 minutes, and an agent that takes no wakes is skipped without writing a row. - Company-wide passes resume after the last stall a capped pass reached. The per-issue sweep also checks the issue's own children and blockers, which covers an unassigned stalled blocker. - The dependency gates no longer trust the reason string. `verifyBlockerStallWake` requires a system request, the stalled issue's current stall action, and a real child or blocker edge. An agent-requested wake with the same context is parked (tested). - The notice shows a refused Retry's reason, and a lost response as "Taking longer than usual…", with Retry still available (tested). Review round 2 (Greptile), each with a regression: - Race: two passes claiming one stall leave one delivered row (unique index; the loser's insert is skipped). - The company recovery sweep selects companies whose only stall is a resolved-`blocked` board action, so an unassigned stalled blocker with a todo waiter is reached. - Private stalled issue: the waiter's wake and activity carry no title, identifier or recovery details. - `GET /issues/:id` always returns `stalledDependencies`, so a cleared stall replaces the cached list. - The notice shows a failed child read with a Reload button (tested). - `pnpm --filter @paperclipai/db check:migrations` passes. Review round 3 (Greptile), each with a regression: - A stall wake that fails to enqueue is retried: the recovery-action pass reports `blockerStallWakeFailures` and the dependency pass counts it in `enqueueFailed`, so the worker throws and retries at its backoff (`issue-recovery-work-lifecycle.test.ts`). - The notice drops a stall once the child's refreshed status has moved on (`StalledDependencyNotice.test.tsx`, 7 tests). ## Risks - A deploy wakes the waiters of every issue that is stalled today, once each. That is intended (they are stranded now), and it's bounded at 200 stalls per pass. - A stall wake may run while blockers are unresolved, like a comment wake. It is bounded to one per stall, and pause holds still apply. - The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet list the new reason. The wake prompt carries the guidance. - One additive index, built concurrently; no persisted run state change. Existing rows cannot violate it before deploy, since no `issue_blocker_stalled:` keys exist yet. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 18:26:07 -06:00
DottaandClaude Opus 5.5	b43cccbb08	fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the runner reports it as `provider_turn_lost_on_restore` and the controller starts one same-conversation continuation. A later restart that loses the continuation is answered the same way > - With back-to-back restarts, the controller's checkpoint can miss a continuation that the runner already accepted. On the next restore the controller starts a fresh continuation, and the restored runner then reports the unrecorded earlier continuation as lost > - The event consumer treated that stale lost terminal as the run's terminal, so the run failed while its new continuation was already running > - Recovery then refused to replay the failed run, and the task needed a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but the parent stayed `blocked` on that subtask > - This pull request makes the consumer skip a restart-lost terminal for any turn other than the continuation it started > - The benefit is that back-to-back restarts no longer turn a recoverable lost turn into a failed run that needs Retry ## Linked Issues or Issue Description Refs #20107 **What happened?** Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade between builds plus repeated restarts) at 070365404e. In task t02 (claude M1), the codex subtask CHA-7 failed with `native_provider_terminal_failed: The runner restored the conversation after process loss, but the previous turn is no longer active.` The sequence: 1. Restore 1 lost turn `01a1bd17…`, and the controller started continuation `01a1bd18-15ec…` (runner command 14). A restart followed within a second, before the controller's checkpoint recorded that continuation. 2. Restore 2 saw the checkpointed interruption of `01a1bd17…` and started a fresh continuation, `01a1bd18-4cc4…` (command 24). The restored runner also reported the unrecorded continuation `01a1bd18-15ec…` as `provider_turn_lost_on_restore`. 3. Because a continuation was already started, the consumer returned that stale terminal as the run's terminal. The run failed, the new continuation was stopped (command 25), and automatic recovery refused to replay it (`uncertain_provider_action`, `preserve_without_replay_v1`). 4. The parent CHA-2 asked CHA-7 to resume by comment, but an agent comment cannot authorize a blocked replay, so the wake stayed `deferred_issue_execution`. The parent's own Retry succeeded, but CHA-2 stayed `blocked` on CHA-7 (`R3:retry_did_not_complete`). **Expected behavior** A restart-lost terminal for an older, unrecorded continuation is superseded by the continuation this restore started. The run continues on that continuation and needs no Retry. **Steps to reproduce** 1. Checkpoint a native session whose last terminal is a restart-lost turn and whose active turn is null, so recovery starts a continuation. 2. Have the restored session report another turn (the unrecorded earlier continuation) as `provider_turn_lost_on_restore`, either before or after the new continuation starts. 3. Before this change, the run ends on that stale terminal. **Paperclip version or commit** 070365404e (wave); branch rebased on current master. ## What Changed - `native-session-runtime.ts`: - Records the turn id of the restart continuation it starts. - New `supersededByRestartContinuation`, used by the consumer's `resumeInterruptedTurn`, keeps consuming past a `provider_turn_lost_on_restore` `turn.failed` when all of these hold: - the run was recovered from a checkpointed interruption; - the failed turn is neither the checkpointed interrupted turn nor the started continuation; - a continuation was actually started. - The check first waits for the startup decision, because the consumer starts before the continuation does. - `native-session-runtime.test.ts`: a red/green test, with the stale lost terminal arriving both after and before the continuation starts. ## Verification - Red/green: - On master, both test cases end the run on the stale terminal. - With the change, both resolve on the continuation's result. The continuation is started once, and the stale `turn.failed` is still appended to the journal. - `vitest run` over the restart, recovery, continuation and terminal test files in `packages/paperclip-runner`, including all of `native-session-runtime.test.ts`: 9 files, 380 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck:typescript` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The skip applies only to restart-lost failures of turns other than the checkpointed interrupted turn and the started continuation, and only after a continuation actually started. Real provider failures, a loss of the continuation itself, and runs without a checkpointed interruption keep their current handling. - If startup fails, the startup decision still settles. The consumer then falls back to the existing behavior and the run fails as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 16:44:13 -06:00
DottaandClaude Opus 5.5	412d49e48d	fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112) ... ## Thinking Path - The rule for upgrade-safe saved state: work saved by the old build must resume under the new build. - The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke that rule for Claude. The same heartbeat run resumed after the upgrade restart, and the runner's resume probe (`runner.drain`) failed with `ACPX durable launch profile digest does not match runner startup` (`provider_state_incompatible`). t03 and t04's Claude subtask both needed a board Retry. Codex crossed the upgrade cleanly. - The ACPX launch profile digest covers the provider-pack authority and the sidecar artifact hashes, so every deploy changes it. - #18879 already re-binds a suspended session to the new digest. A deploy that restarts the runner mid-turn leaves the same run with an active turn, or with an open session, which is the state a same-build lost-turn restore leaves behind. Neither was re-bound. - A same-build restart already recovers both states safely: it reopens an open session, and recovers an active turn through `restore_conversation_after_lost_turn`, which is guarded by the provider-lifetime fence. An upgraded build should take the same path. ## Linked Issues or Issue Description Refs #18879 **What happened?** A deploy restarted the runner while a Claude (ACPX) task was mid-run. When the same run resumed under the new build, restore refused the saved state with `provider_state_incompatible`. The task stopped and needed a manual Retry, which started a fresh provider session and dropped the conversation. **Steps to reproduce** 1. Start a Claude task on build A and let it reach an active turn. 2. Upgrade to build B, which ships a new provider pack or sidecar build, with a graceful restart while the turn runs. 3. The same run's resume fails with `ACPX durable launch profile digest does not match runner startup`. **Expected behavior** The resumed run recovers exactly as it would after a same-build restart. The conversation is restored, or the lost turn is closed through the normal recovery path, with no Retry. ## What Changed - `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`: - `rebinds_settled_session` now also re-binds the saved digest for an **interrupted turn or open session of the same heartbeat run** (`rebinds_interrupted_turn`). - The re-bind requires exactly the same-build recovery preconditions: no accepted semantic result, no pending runtime request, no unconfirmed provider exit, and a dormant goal. It also keeps the existing requirements: no candidate admission, and the same sidecar command and arguments. - Recovery then runs unchanged. The provider-lifetime fence still decides whether the conversation is restored or the lost turn is closed. - **Relocated installs:** an upgrade can install the same launch files under a new directory, as an `npx` versioned cache, a new image layout or a new checkout does. The K14 harness moves between worktrees. The re-bind now accepts a launch of the same shape: the same executable and sidecar file names, and identical non-path arguments. It then records this startup's runner-owned paths in the saved descriptor. A renamed executable or a changed argument stays strict. - An active turn or open session saved by a *different* run stays strict, as before. So do suspended sessions with undelivered events, and every qualified-release binding (ACPX, agent server and command digest). - Tests (Rust, `acpx_provider_backend`): - `interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`: red on master. An old-digest saved state with an active turn restores, the digest is re-bound, and identity and command digest are kept. - `open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`: restore gets past the saved-state check, to provider start. - `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`: red on master and on the first fix, with the production error. It covers an interrupted turn of the same run and a settled session attached by the next run, both moved to a new install directory. - `interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`: each unsafe guard still fails with `provider_state_incompatible` and leaves the saved state untouched. - The existing strict tests for other runs are unchanged and still pass. - `packages/paperclip-runner/docs/durable-recovery.md`: documents the extended re-bind. - `tests/acpx_warm_attachment.rs`: the end-to-end test `restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`, from the Phase 2 session's parallel fix. It fails on master with the production error and passes here. ## Verification - `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44 passed. - The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures are timing-based `task_memory` kill-monitor tests on macOS; that file is untouched (last changed 5 days ago). CI runs them on Linux. - `cargo fmt --check`: clean. - Live: the first K14 rerun, at 070365404e, still failed for 3 Claude runs. The cause was the install path: the saved sidecar path pointed into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun has been requested. ## Risks - **The restored conversation runs on the new provider pack.** That is already the case for re-bound suspended sessions. Session start still verifies every artifact and the restored provider identity against the qualified command digest. A changed upstream release still fails strict. - **Old provider process.** For an active turn, lost-turn recovery waits for the provider-lifetime fence before restoring. For an open session, behavior matches the same-build restart, which reopens without the fence. - **A state that really is incompatible** (an upstream provider release change): - **Between turns**, the common upgrade case: the next run's checkpoint carries `allow_replacement_after_resume_failure`, so the refusal opens a fresh provider session automatically, with no Retry. A new test, `continues in a fresh session when a new run's saved session is incompatible…`, pins this in `native-session-runtime.test.ts`. - **Mid-turn, same run:** this still stops for a board Retry, by design. The effects of the lost turn are unknown, and the automatic replacement sweep (`decideNativeReplacement`) only replaces a session when it has a complete effect inventory. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 16:34:23 -06:00
DottaandClaude Opus 5.5	15b987d9b8	fix(runner): a finish or block waits for held calls before it is judged (#20108) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restarts must be invisible to a task: an agent that was working keeps working after the control plane comes back > - Restart chaos t05 (Daytona warm wave, codex, K4 hold) ended a task as blocked even though its document save succeeded > - #20069 refuses a finish or block that the agent decided before it saw results that had already returned. Here the results had not returned yet when the block was checked: after the reattach, the held calls and the later block are handled at the same time > - This pull request makes a finish or block wait, for a bounded time, for earlier calls of its turn that have not returned, and then apply the same check > - The benefit is that a controller outage during a Paperclip tool call no longer turns into a false "blocked" task ## Linked Issues or Issue Description Refs #19831 (durable ownership plan). Restart chaos wave `w-20261108T192459-s9501`, task t05 (Daytona warm, codex, C1, K4 hold of 182 s, master c5a172a183, which includes #20069). **Problem** 1. 19:37:01: the follow-up comment arrives. The agent reads the brief and calls `write_document`. 2. 19:37:44 to 19:40:51: K4 hold. Codex keeps the call pending and re-samples the model: "The save request is still pending", then `read_document` (also pending), then `paperclip_block`. 3. 19:42:02, after the reattach: the controller recovers the held write and read (`recoverPendingSemanticInputs`) while the Runner delivers the block. The block reaches the check at seq ~331–334. The write's result, with the FOLLOWUP line saved, is committed at seq 337. The check saw no returned call, so the block was recorded and the issue rested `blocked` with no Retry. So held calls are answered after ready (the save succeeded, the brief has the marker). The defect was the order in which the terminal call was judged. ## What Changed - `packages/paperclip-runner/src/live/terminal-call-freshness.ts`: - `pendingEarlierToolCalls`: calls made before the terminal call that have neither returned nor reached the agent; - `settleEarlierToolCalls`: waits, bounded at 30 s (`TERMINAL_CALL_SETTLE_TIMEOUT_MS`), until none is pending, then reports which returned unseen and which are still pending; - `staleTerminalCallResult` also names still-pending calls: "has not returned yet. Wait for its result (do not repeat a write until it returns)…", still `terminal_call_before_results`, retryable. - `runnerd-codex-transport.ts`: before dispatching `paperclip_finish` or `paperclip_block`, waits for those calls and re-checks session identity. It refuses the terminal call if any earlier call returned unseen or is still pending. - Scope: `DurablePrpCodexTransport` serves Codex and ACPX on local, warm Daytona and per-turn Daytona, so every path that reattaches to a surviving Runner is covered. The wait applies only to terminal calls with earlier unreturned calls, so it adds nothing before `turn.start`. - Not covered: the external-driver path (`drivers/external/runnerd-external-driver.ts`) has its own semantic handler without either check. It is recorded as a campaign follow-up. ## Verification Red/green: - `runnerd-codex-transport.test.ts` "refuses a terminal call that arrives before a held call's result (restart chaos t05)": the write is started without a result, the block arrives, and the write's `semantic_tool.result` lands 100 ms later. - On master's transport and guard: fails, the block is dispatched (`handler` called once). - With this PR: the block is refused with `terminal_call_before_results`, naming the returned write. - `terminal-call-freshness.test.ts`: pending versus returned, a result that lands during the wait, a call still pending after the bound, no wait when everything was seen, and a session that goes away. - Both files: 300 passed. `tsc --noEmit -p packages/paperclip-runner` is clean. Live: the crash point `runner.semantic_tool.after_input_before_dispatch` (from #20069) holds K4 during a Paperclip tool call. A Daytona run against this branch is requested from the slot owner after merge. ## Risks - A finish or block made in the same model step as a write (parallel tool calls) now waits for the write and is refused once, because the agent had not seen the write's result. The agent calls it again after reading the result: one extra model step in that case. #20069 already behaved this way whenever the write returned first. - A Paperclip call that never returns delays a terminal call by 30 s, and the terminal call is refused. Recovery answers held calls after ready or disconnects the Runner, so the refusal cannot repeat indefinitely in practice. - No schema or persisted run-state change. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above (#20069) - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments; no user docs affected) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 16:08:06 -06:00
DottaandClaude Opus 5.5	a8bbdf237c	fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Claude ACPX run must survive restarts and suspensions. A turn stopped for suspension drops the live ACPX session and leaves the saved lifecycle `prepared` > - The next `turn.start` checked for a live session before anything else, so it failed with "ACPX session is not open". That happened even though the saved session was safe to reopen > - In restart chaos this made a Claude approval task (G1 t03) need a Retry after repeated restores > - This pull request makes `turn.start` reopen a reopenable session before it starts the turn > - The benefit is that a suspended or restored Claude run keeps going on its own, with no Retry ## Linked Issues or Issue Description Refs #20089 **What happened?** In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed after repeated restores with `ACPX session is not open` and needed a Retry. The Retry overwrote the session directory, so the exact sequence could not be read back. The same error reproduces directly with `turn.stop` (suspension) followed by `turn.start` on the same session: the stop clears the live session and sets the lifecycle to `prepared`, and `turn.start` never reopens it. **Expected behavior** When the saved session can be reopened, `turn.start` reopens it, the same way `session.open` does, and then starts the turn. "Can be reopened" means: the conversation identity is recorded, the provider exit is confirmed, the session belongs to this run, and the lifecycle is `prepared`, `suspended` or `session_open`. A session that was closed on purpose, or whose provider exit is unconfirmed, still fails. **Steps to reproduce** 1. `run.prepare`, `session.open`, then `turn.start`. 2. `turn.stop` (suspension stop). 3. `turn.start` again. Before this change, it fails with "ACPX session is not open". **Paperclip version or commit** master at f789f3e2af. Branch rebased on current master. ## What Changed - `acpx_provider_backend.rs`: `turn.start` checks whether there is no live session and the saved state is reopenable. If so, it reopens through `open_session` first, and the reopen's `session.resumed` event comes before the turn's events. The previous body moved unchanged into `start_turn_in_open_session`. - `tests/acpx_warm_attachment.rs`: a new test, `reopens_the_conversation_for_a_turn_after_a_suspension_stop`. It drives prepare, open, turn, stop, turn and expects the second turn to be accepted with `session.resumed`. It also checks that after `session.close`, `turn.start` still fails. ## Verification - Red/green: the new test fails on master with "ACPX session is not open" and passes with the change. `cargo test -p runner-core --test acpx_warm_attachment`: 3/3 passed. - I also ran the full `runner-core` crate suite. Five tests failed: four `task_memory` monitor tests and `unconfirmed_suspension_state...`. Each of them passes on an unchanged master checkout, and the ACPX one also passes when rerun in this branch, so they are timing-dependent and not caused by this change. ## Risks - Low risk. A reopen happens only under the same conditions in which `session.open` would already succeed for this run. A closed session and an unconfirmed provider exit stay fail-closed. The order of events changes only by adding `session.resumed` ahead of the turn's events. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 15:46:11 -06:00
DottaandClaude Opus 5.5	a88e23fa34	fix(runner): read an uncommitted turn start as empty history when the event window is complete (#20107) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run must survive repeated control-plane restarts; each restore reconciles the turn its checkpoint names as active > - That reconciliation reads the turn's history. When the provider has no paginated history, the runnerd transport rebuilds it from the committed runner events > - After back-to-back hard restarts, the checkpoint can name an accepted turn whose start was never committed. The runner was killed after journaling `turn.start` and before the provider emitted anything > - The rebuild treated every missing start as lost history and failed with `codex_history_incomplete`, so the task needed a Retry > - This pull request treats a missing start as empty history when the committed window is provably complete > - The benefit is that a run survives back-to-back restores, as it already survives one ## Linked Issues or Issue Description Refs #20074 **What happened?** In restart-chaos wave w-20261108T200620-s9701 (local, K4/K6 + M1, back-to-back K4 restores), three runs failed on the second restore. Two were Claude ACPX C1 tasks; the third was an M1 parent with its subtask. They failed with `native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/items/list requires supported paginated history; Error: codex_history_incomplete: requested turn start is outside the retained runner event window`, and each needed a Retry. The sequence: 1. The first restore lost the in-flight turn and started a continuation turn. ACPX echoes the requested turn id, so the checkpoint named it as active. 2. Six seconds later a second K4 SIGKILL killed the runner. The runner had journaled that `turn.start` (now `execution_indeterminate`), but the provider had emitted nothing, so no `turn.started` was committed and the runner outbox was empty. 3. On the second restore, `thread/items/list` for that turn hit the guard, although the committed window was complete: source sequences 1–25, nothing evicted. **Expected behavior** The second restore treats the never-started turn like the first restore treated its lost turn. The runner reports it lost on restore, and recovery starts a continuation with no Retry. **Steps to reproduce** 1. Have a checkpoint name an accepted turn whose `turn.started` was never committed. For example, kill the runner after `turn.start` and before the provider emits anything. 2. Restore the session. Before this change, recovery fails with `codex_history_incomplete`. **Paperclip version or commit** master at 88835e624a plus #20094 (the wave). Branch rebased on current master. ## What Changed - `runnerd-codex-transport.ts`: a new `committedEventWindowComplete(state)`. It is true only when the committed events are exactly source sequences 1 to `ackedSourceSeq`: none missing, none duplicated, none evicted. - In the `thread/items/list` fallback, a turn with no committed start returns empty history when the window is complete. Otherwise it still throws `codex_history_incomplete`. ## Verification - Red/green connected test in `runnerd-restart-continuation.test.ts`: "starts a new turn when the checkpoint names an accepted turn whose start was never committed". It uses the real runnerd and the fake Codex server, and holds the provider start with `--turn-started-gate`. - Without the change, it fails with the exact production error. - With the change, the run completes on a continuation turn. - Unit test: `committedEventWindowComplete` is false for a gap, a missing first event, duplicates and no state. - `npx vitest run` over every Codex, restart and recovery test file in `packages/paperclip-runner`: 45 files, 1158 tests passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`packages/paperclip-runner`): passed. ## Risks - Low risk. The empty history applies only when every acknowledged event is still retained, so no event of that turn can have existed and been dropped. Any evicted or gapped window keeps failing closed. - Known gap, recorded as a follow-up: if eviction did remove events of a turn the checkpoint still names, recovery still fails closed. The planned fix is to never evict events of an unresolved checkpointed turn. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 15:37:11 -06:00
DottaandClaude Opus 5.5	3bf51913ef	fix(daytona): gzip transport compression for sandbox images without zstd (#20104) ... ## Thinking Path - Per-turn remote sandboxes are the default, so every remote turn uploads the workspace into a fresh Daytona sandbox. - The Daytona plugin compresses large uploads with zstd, but only when the sandbox has a `zstd` binary. Without one, the upload goes raw: 220 MB for the paperclip seed. - The default Paperclip runner image has zstd, so it is already covered. A sandbox image without zstd is not: for example a user-supplied `config.snapshot` built on plain `debian:bookworm`. Such an image still has gzip. - This adds gzip as a fallback codec for those images. It is a robustness fallback, not a speedup for the default image. ## Linked Issues or Issue Description Refs #20100 **What happened?** On a Daytona sandbox image without a `zstd` binary, every remote turn uploads the workspace seed uncompressed. Measured live with the paperclip seed: raw takes 18.6–28.8 s, against 6.1–16.0 s compressed. **Steps to reproduce** 1. Use a Daytona snapshot whose image has gzip but no zstd. 2. Run a remote task on a large repository and inspect the `transfer` span: there is no `transfer.compression.codec` attribute, and the full raw size is uploaded. **Expected behavior** Large workspace tarballs are compressed in transit whenever the sandbox can decompress them. ## What Changed - `packages/plugins/sandbox-providers/daytona/src/file-sync.ts`: - The existing availability probe (the same round trip as `mkdir`) also reports `gzip`. - Codec choice: zstd when the sandbox has it and Node supports it (unchanged), otherwise gzip, otherwise raw (unchanged). - gzip runs at level 1. On the 220 MB paperclip seed it produces 69.6 MB in 1.26 s on the host; zstd -3 produces 57.6 MB in 0.54 s. - In the sandbox, gzip decompresses with `gzip -dc -- <scratch> > <raw scratch>`. It reuses the existing explicit `chmod`, atomic `mv -f`, failure sweep, private host temp directory, and minimum-saving fallback. - A failed or partial decompress exits 49, the raw scratch is never promoted, and every scratch file is swept. The workspace sentinel and `baselineSha256` checks still run before `turn.start`. - The `transfer.compression.codec` span attribute is now `zstd` or `gzip` (still a closed set). - Tests in `file-sync.test.ts` (red on master): - A real shell promotion on a sandbox that reports no zstd: the upload is `.gz` and under half the size, `gzip -dc` runs, and the target is byte-identical with mode 0600 and no scratch left. - A truncated gzip upload fails the sync, never creates the target, and leaves no scratch. - Codec preference: zstd when both are present, gzip when only gzip is, raw when neither is. ## Verification - `npx vitest run` in `packages/plugins/sandbox-providers/daytona`: all pass. The existing zstd suite runs on this host because it has zstd. - `tsc --noEmit -p packages/plugins/sandbox-providers/daytona`: clean. - Live Daytona run on the runner image, 2 vCPU, 220 MB seed, through the real `performSyncIn` with each codec forced through the probe. The sha256 matched every time. | Codec | Times | |---|---| | raw | 28.8 s, 18.6 s, 20.2 s, 22.8 s | | zstd | 6.9 s, 6.1 s, 16.0 s, 8.5 s | | gzip | 9.8 s, 16.8 s | Network variance is large. Both codecs roughly halve the raw upload or better. ## Risks - **Host CPU:** about 1.3 s of streamed gzip per large upload, on the libuv thread pool, and only when the sandbox lacks zstd. Files under 8 MB, or that save less than 10%, still go raw. - **No compressor at all:** a sandbox without gzip or zstd uploads raw, as today. - **Probe caching:** the probe runs on every sync, on the `mkdir` round trip it already makes, so it adds no latency. It stays uncached, following the existing design's rule that one principal's observation is never reused for another. - **Observability:** one new value, `gzip`, in an existing closed-set span attribute. The endpoint gate is unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 14:52:56 -06:00
DottaandClaude Opus 5.5	015d3d0a7c	fix(environments): tear down retain-on-failure sandboxes when their environment is deleted (#20085) ... ## Thinking Path - Remote sandboxes cost money while they exist, so every sandbox Paperclip keeps must have an owner and an end. - Two lease policies keep a sandbox after its run. `reuse_by_environment` keeps a warm sandbox for the next turn. `retain_on_failure` keeps a failed run's sandbox, stopped, as a workspace-restore source. - Environment deletion is the last chance to end both: with the operator's consent it destroys the environment's kept sandboxes, otherwise it refuses. - It only looked at `reuse_by_environment`. A `retain_on_failure` sandbox neither blocked the delete nor got destroyed, so it was orphaned the moment its environment was deleted. ## Linked Issues or Issue Description Refs #20048 **What happened?** In Daytona warm wave w-20261108T170423-s9501, the harness deleted the environment after grading. Three stopped sandboxes (4e20c8b3, abf0980b, 5e49c561) remained afterwards and had to be deleted by hand through the Daytona API. All three belonged to `retain_on_failure` leases: failed runs' sandboxes kept as workspace-restore sources. **This leaks on master today, with per-turn sandboxes too.** The four leases in that wave carried `workspaceRestoreRecovery` from `legacy-workspace-restore-recovery.ts` (no `nativeSourceRetention` marker). That writer turns any failed run's lease with a workspace-restore failure into a `retain_on_failure` restore source, whether the environment is warm or per-turn. Every such sandbox has been surviving environment deletion and costing money until deleted by hand. **Steps to reproduce** 1. A run in a sandbox environment fails, and its lease is released as a `retain_on_failure` restore source (sandbox stopped, not destroyed). 2. Delete the environment with `destroyReusableSandboxLeases=true`. 3. The delete succeeds; the restore-source sandbox is never destroyed and has no lease or environment left to find it by. **Expected behavior** The delete blast radius counts kept restore-source sandboxes like reusable ones. Without consent, the delete is refused; with consent, they are destroyed through the same claimed teardown before the environment row is removed. ## What Changed - `server/src/services/environments.ts`: `getDeleteBlastRadius` counts leases with policy `reuse_by_environment` **or `retain_on_failure`** (status `active`, `released`, `retained` or `failed`) as kept-sandbox holders. The blocked reason is still `reusable_sandbox_lease`. - `server/src/services/environment-runtime.ts`: `destroyReusableSandboxLeasesForEnvironment` selects both policies. It claims through `destroyClaimedSandboxLease` with `anyLeasePolicy` plus a scope condition limited to those two policies, keeping its existing guards: live-run skip, pending export, legacy shutdown protection, and the claim compare-and-swap. - `server/src/__tests__/environment-runtime.test.ts`: a released `retain_on_failure` lease on a failed run now blocks the delete and is destroyed by the consented teardown (`expired`, `environment_deleted`). This fails on master. ## Verification - `cd server && npx vitest run src/__tests__/environment-runtime.test.ts src/__tests__/environment-service.test.ts src/__tests__/environment-routes.test.ts`: 347/347 pass. - The new test fails on master: the blast radius reports no kept sandbox. ## Risks - **A delete now refused that used to succeed.** An environment holding restore-source sandboxes is refused until the operator consents to destroying them. That is the intended behaviour; it was silently leaking them before. - **Losing restore files.** The consented delete destroys restore sources, which is what the operator agreed to by deleting the environment. Leases with a pending export or legacy shutdown protection are still skipped and keep blocking. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 12:32:38 -06:00
DottaandClaude Opus 5.5	a9ae1b69c3	fix(runner): let a finish through after a restart lost the provider turn (#20088) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner relays an agent's Paperclip tool calls and keeps them across control-plane restarts > - #20069 refuses a `paperclip_finish` or `paperclip_block` decided before an earlier returned result reached the agent > - The guard filters by turn id, but each committed event carries the run's one identity turn id, so provider turns are not separated > - When a restart loses a provider turn, a call of that turn gets its result but never its provider completion > - The guard then refuses every later finish of the run, so the run exhausts its retries and the user must click Retry > - This pull request makes a new provider turn end the pending calls of the previous turn > - The benefit is that an agent can finish after a restart that lost its turn ## Linked Issues or Issue Description Refs #20069 **What happened?** A restart chaos storm (local, Codex, A1/C1, K2 drains for 30 minutes, 116 tasks) had one failure. Task t110 needed a user Retry. Its run was drained twice. The first drain lost the provider turn while a `read_agent_instructions` call was pending. After that, all four `paperclip_finish` calls failed with `terminal_call_before_results` and named `read_agent_instructions`. The run ended with `native_session_retry_exhausted`. **Steps to reproduce** 1. Start a Codex run that calls a Paperclip tool. 2. Drain the control plane while that call is pending, so the Runner restores the session and reports `provider_turn_lost_on_restore`. 3. Let the agent continue in the new turn and call `paperclip_finish`. **Expected behavior** The finish is recorded. The lost call belongs to a turn that the agent will not see again. **Actual behavior** The Runner refuses every finish with `terminal_call_before_results`. The run fails and needs a user Retry. ## What Changed - `unseenReturnedToolCalls` forgets the calls it tracked when a `turn.started` event begins a new provider turn. - A new test reproduces the t110 event order: a call in a lost turn, then a new turn that reads, sees its result, and finishes. ## Verification - `npx vitest run src/live/terminal-call-freshness.test.ts` in `packages/paperclip-runner`: the new test fails before the change and passes after it (7/7). - `npx vitest run src/live/runnerd-codex-transport.test.ts -t terminal`: 10 passed. - `tsc --noEmit -p packages/paperclip-runner`: clean. - Evidence: the kept Runner control-plane state of t110 shows the call `exec-0f3d1b90` (generation 1) with a `semantic_tool.result` command but no `tool.execution.completed`, followed by `turn.failed provider_turn_lost_on_restore` and new `turn.started` events. ## Risks - The t03 case of #20069 stays covered: the stale block and the earlier calls are in the same provider turn, with no `turn.started` between them. The existing t03 tests still pass. - If a driver emits `turn.started` inside one provider turn, the guard can miss a stale call of that turn. No current driver does that. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 12:29:38 -06:00
DottaandClaude Opus 5.5	4f7eee10c2	feat(recovery): let the board confirm an unrecorded owner stopped for every settling process kind (#20084) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery releases in-progress records (leases, claims, intents) only when it can prove their owner is gone > - A record saved before its server process was fully recorded is `unprovable`, and it is never released on its own > - #20067 added a board-only action to confirm that such an owner stopped, but only the environment run lease used it > - Other settling kinds (for example the legacy controller lease, which shows `missing_identity` rows in the Phase 2 reconciler data) stay stuck with no way forward > - This pull request opens the same confirmation to every settling kind whose record names a partly recorded server process > - The benefit is that an operator can release these stuck records safely, and recovery settles them at once ## Linked Issues or Issue Description Refs #20067, #20054 **What happened?** The Phase 2 reconciler data shows `legacy_controller_lease` records that stay `unprovable` with `missing_identity`: they were saved before Paperclip stored the controller's PID space. Recovery never releases an unprovable owner, and only the environment run lease offered the board's "confirm the previous server stopped" action. These records, and records of other settling kinds saved the same way, had no path forward. Also, a board confirmation did not wake the reconciler. Unprovable records are not watched, so the release waited for some unrelated pass. **Expected behavior** Recovery asks the board on the record's task. After the board confirms, recovery releases the record through the kind's own compare-and-swap settle, at once. A confirmation never releases a record that changed since the board was asked. **Steps to reproduce** 1. Have a legacy run whose `controller_process` was recorded without a host hash (an older server). 2. Run the ownership reconciliation pass. The record is `unprovable` (`missing_identity`) and no board action appears. **Paperclip version or commit** master at c55af26821. ## What Changed - `ownership-registry.ts`: a new `ownerGoneAttestable` helper opts a settling kind in. The kind asks the board (cause `ownership_owner_unprovable`) only for a process-owned record that is unprovable for a legacy reason and has a version. Opted in: `legacy_controller_lease`, `environment_teardown_claim`, `sandbox_creation_intent`, `workspace_runtime_control_operation`, `native_workspace_finalize_attempt`, `native_finalizer_lease`, `native_cleanup_lease`. - `owner-gone-attestation.ts`: - The request records the record version (`recordUpdatedAt`), and each pass refreshes an open request to the current version. - The board submits the version it was shown (`recordVersion`, a new optional field in `confirmOwnerGoneSchema`). The confirmation is refused with 409 unless that equals the request's current version and the locked record's version. - `recordVersionAttestations` reads back every resolved confirmation. Any one that names the record's current version applies. - `confirmOwnerGone` now wakes the ownership reconciler in its transaction. - Each opted-in kind has a version lock. For an `operator_attested` proof, its settle runs in a transaction that locks the record row and re-checks the confirmed version before the kind's own compare-and-swap. - `ownership-invariant-report.ts`: in the periodic pass and in one-shot settles, a matching confirmation turns a legacy `unprovable` verdict into `gone` (`operator_attested`). A live or provable owner always wins. The extra read runs only when a kind has such a record. - `WorkspaceExportRecovery.tsx`: the board's confirmation names the stuck record by kind (for example "This task's earlier run"), with a generic fallback, and sends the version it showed. - `doc/execution-semantics.md`: - It describes the action, the version binding and which kinds are offered. - It updates the controller-lease section to match #20054: the holdings list, and that the lease is off only on Railway deployments with an exclusive volume. ## Verification - `pnpm exec vitest run src/__tests__/owner-gone-attestation.test.ts` (server): 10/10. The new legacy-controller cases fail without the change: - the board is asked, and nothing is released before the confirmation; - the confirmation wakes the reconciler, and the next pass settles the record as `operator_attested`; - a record that changed after the confirmation is not released, and a fresh request is raised; - a record of these kinds with no owner stamp is not offered, because its settle could never release it; - a confirmation for a version the record no longer has is refused, and the refreshed request then applies; - a write between the reconciler's listing and its settle stops the release inside the settle's transaction. - The 10 server ownership suites (registry, report, owner proof, controller lease, legacy controller, teardown, finalize, token leases): 332/332. - `pnpm exec vitest run src/components/WorkspaceExportRecovery.control.test.tsx` (ui): 48/48. - `pnpm check:token-gates`: all gates clean. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` and `ui`: passed. ## Risks - A wrong release would be serious. The safeguards: - The confirmation applies only while the record's last-write time equals the one the board was asked about, and only while the prover still finds the recorded process unprovable for a legacy reason. - Each kind's settle still compare-and-swaps on the exact recorded process. - A record of these kinds with no owner stamp at all is not offered. Those settles cannot release it, so it keeps its existing recovery path. - Kinds that are unowned by design (queued pending cleanup, accounting debt) are not offered. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 12:16:07 -06:00
DottaandClaude Opus 5.5	c55af26821	fix(runner): treat a replayed start of a turn lost on restore as history (t48) (#20074) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run must survive a control-plane restart and continue its work > - #20057 made recovery adopt an in-flight turn, or start a new turn when the restored runner reports the old turn lost > - One case was not covered. A graceful drain can detach the controller after the runner accepted `turn.start` but before the provider started the turn > - The runner then records the provider's `turn.started` while no controller is attached. After a restart it replays that start, and the Codex session rejects it as an unexpected turn > - This pull request treats that replayed start as history from the earlier generation > - The benefit is that the run starts its continuation turn instead of stopping with `native_event_replay_conflict` ## Linked Issues or Issue Description Refs #20057 **What happened?** A restart chaos wave ended with one failed task (t48, Codex). A graceful SIGTERM suspended the controller 300 ms after the runner accepted `turn.start`. The provider's `turn.started` came after the controller detached. Recovery classified the run as `resume_dead_runner`. The restored runner reported the turn as `provider_turn_lost_on_restore`, which is correct. But it also replayed the undelivered `turn.started`. The Codex session failed with "Provider started an unexpected turn." (`turn_binding_mismatch`), and recovery stopped with `native_event_replay_conflict`. **Expected behavior** Recovery starts a new continuation turn. The replayed start of the lost turn is recorded as history, and the run completes. **Steps to reproduce** 1. Start a Codex native run. 2. Detach the controller gracefully after the runner accepts `turn.start` and before the provider emits `turn/started`. 3. Let the runner record `turn.started` while it is unattached, then kill the runner and the provider. 4. Restore the run. Before this change, the session fails with `turn_binding_mismatch`. **Paperclip version or commit** master at a9cf4ee731 (the wave ran on 66a2cbab98). ## What Changed - `codex-app-server-driver-impl.ts`: recovery identifies the turn that this run's latest `turn.start` had the runner accept, when recovery did not adopt it and provider history does not show it live. Recovery passes it to the session as `lostAcceptedTurnId`. - `codex-session-state.ts`: the session keeps `lostAcceptedTurnId` as an unconfirmed lost turn. The stale-event rule treats its replayed start and items as history. - `codex-session-notifications.ts`: `turn/started` applies the stale-event rule to earlier-generation and unconfirmed lost turns. The runner's restart-loss terminal confirms the lost turn. Any other terminal, or a semantic result, from that turn fails closed, so a real outcome is never dropped. A start for a turn that this generation settled, or for a turn the session never saw, still fails closed. - `runnerd-restart-continuation.test.ts`: a new connected test with the real runner and the fake Codex server reproduces the t48 shape. A new fake option, `--turn-started-gate`, holds the provider's start until the test detaches the controller, so the test does not depend on timing. - `codex-app-server-driver.terminal-replay.test.ts`: driver tests for the replayed start (history), and for an ordinary failure or a result from the lost turn (fail closed). ## Verification - Red: before the fix, the new test failed with `NativeProviderTerminalFailure: Provider started an unexpected turn.` at `codex-harness-session.ts:211`. This is the same error and stack as the chaos run. - Green: `npx vitest run src/drivers/codex src/live/runnerd-restart-continuation.test.ts` (in `packages/paperclip-runner`) passed 392 of 392. - The wider Codex, restart and recovery test files (45 files, 1153 tests) passed on the final revision. The two fail-closed driver tests fail on the first revision, where the lost turn's outcome was dropped as history. The one failure in an earlier run ("still rejects history for a turn that became terminal in the live session") showed that the first version was too broad. The rule now applies only to earlier-generation turns, and that test passes. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `packages/paperclip-runner` passed. ## Risks - Low risk. The change only applies after a restore, to a turn that the runner accepted for this run's latest `turn.start`, and only when recovery did not adopt that turn and provider history does not show it in progress. - A turn that the session never saw still fails closed. A re-start of a turn this generation settled still fails closed. A real failure or a result from the lost turn fails closed instead of being dropped. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 11:14:24 -06:00
DottaandClaude Opus 5.5	e8a3f1f6f2	feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, the control plane settles in-progress records (leases, claims, intents) only once it proves their owner process is gone > - The proof compares host and PID space, so a stamp from a replaced container or a rebooted machine stays unprovable forever, and its records never settle > - Self-hosted users on Docker, Kubernetes and Fly hit this on every redeploy; a Railway deployment with an exclusive volume is covered by `deployment_replaced` (#20026) > - This pull request adds a fenced controller epoch: each boot holds a database lease, and an expired boot is fenced so its records can be proven gone > - The benefit is that work stranded by a replaced container is recovered, without keeping an idle database awake ## Linked Issues or Issue Description Follow-up to #20026 and the Phase 2 durable-ownership data. Related: #19965 (owner stamps), #19883 (process space). No duplicate PR was found. **What happened?** A record stamped by a server process in another container reports `different_process_space`. A dead PID elsewhere is never proof, so the record stays `unprovable` and is never settled. Every redeploy of a self-hosted container leaves such records behind. **Expected behavior** A record owned by a server boot that was replaced (stopped cleanly, or lost its lease and was fenced) is proven gone and settled exactly once. A live boot's records stay with it. An idle server issues no recurring queries. **Steps to reproduce** 1. Run Paperclip in a container and start work, so an environment lease is stamped by that boot. 2. Replace the container. The new container has a new PID namespace. 3. The ownership reconciler reports `different_process_space` (unprovable) for the old lease, forever. **Paperclip version or commit** `master` before this PR. ## What Changed - New `controller_instances` table (migration 0449): boot id, epoch (bigserial), process space, pid, a lease on the database clock, `fenced_at` / `fenced_by_epoch`, `stopped_at`. - `controller-instance-lease.ts`, registered with the shared delivery-work coordinator (`controller-instance-lease` queue, runs during drain): - **Holds work → renews; idle → silent.** One upsert creates the row on first use and extends the lease only if the boot holds work: a stamp within the last 90 s, or a stamp whose ownership write may still be pending (held until the server's tracked work settles, capped at 10 minutes; a renewal that sees some other held record does not clear it, so a slow ownership write cannot be outlived), or an in-progress record held by its boot id. Those records are active environment leases, in-flight cleanup claims, sandbox creation intents, workspace finalize attempts, agent lifecycle steps, native finalizer and cleanup leases (`lease_holder` bound to the token), open native coordinator claims, and running legacy runs. Renewal runs every 30 s while the boot holds work. When it finds none, the lease lapses and the worker sleeps until the next stamp (`onControllerStamp`). - **No idle trailing statements.** When the server's last tracked work settles, the worker re-checks holdings at once, at most every 5 s (the rate limit also absorbs the worker's own run). So the last check happens with the work, not 30 s later. - **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or timed-out renewal applies an internal task drain, so no new work is dispatched. A recovered renewal resumes. The boot exits only after 60 s of failure (counted from the first failed attempt's start) while it still holds work, or at once when it finds itself fenced. - **Fence on demand, by a newer epoch.** For a stamp from another host or PID space, the prover reads that boot's row: - `stopped` or `fenced`: gone. - live lease: alive. - otherwise, the boot's last sign of life is the later of its lease expiry and the record's last write plus the TTL. Within 30 s of that, the owner stays `alive` (`controller_lease_grace`), so the settle watch keeps rechecking. After that, the prover fences the boot, but only with a lease row of its own (created on demand) and only an older epoch. - A graceful shutdown records `stopped_at`. - A reconciliation pass that settles nothing (`record: false` or `settle: false`) never fences and never creates the boot's row. - `owner-proof.ts`: these proofs apply only to server-held records. A stamp with a recorded process group stays unprovable, because agent processes can outlive their server. The reconciler passes each record's `updatedAt` to the proof. - `task-admission.ts`: - the quiesce drain never replaces an operator's drain, and has no idle owner, so HTTP stays open; - a `subscribeIdleWorkSettled` hook fires when the last tracked work finishes. - Off on a Railway deployment with an exclusive volume (it has its own replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off` overrides this. A stateless Railway service keeps the lease. A server without its own lease row never fences. - `maintenance-server-routes.test.ts`: in the post-run quiet window, the lease may check once more that the run's records were released. The test now excludes those statements, then asserts that the lease issues nothing in a further window. - Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the two known gaps. ## Verification - `controller-instance-lease.test.ts` (15 cases): - a replaced container with all Railway variables empty is fenced and settled exactly once over two passes; - a live lease is alive; - an owner in grace, and work written after an idle lapse, stay alive and watched; - an older epoch never fences a newer one; - no fencing without its own lease row (off or unregistered), and an idle prover creates its row on demand; - a stopped boot is gone; - process-group stamps stay unprovable; - idle issues zero statements, and one upsert renews only while records are held; - **a 45 s outage:** quiesced, no exit, resumes; - **a renewal that never answers:** bounded, and the boot exits on time; - **a 70 s outage:** exits at 60–65 s, and the successor recovers the work exactly once; - fenced: stops; - the settle re-check; - graceful stop; off on a volume-exclusive Railway deployment, on for a stateless one, and the on/off override; - a running legacy run keeps the lease live. - Red, with the proof's instance check removed: the replaced-container, live, stopped and successor cases fail. - Wider: task-drain, idle-sleep, idle-local-work, owner-proof and ownership suites, plus the maintenance quiet-window tests. Server low-memory typecheck is clean. ## Risks - **Clock skew.** The proof compares a record's `updatedAt` (often the writer's clock) with the lease (database clock). The 30 s grace absorbs ordinary skew. - **A new exit path exists.** A server whose database is unreachable for more than 60 s while it holds work now exits, after first quiescing. This is the intended safety backstop. - **Bounded trailing activity.** A lease that holds work issues one statement every 30 s, plus at most one check when its work settles. An idle server issues none. - **Boots from before this release** have no row, and their records stay unprovable. The board "mark owner gone" action for those follows in a separate PR. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 11:06:32 -06:00
DottaandClaude Opus 5.5	f679cf81dc	fix(runner): refuse a finish or block decided before earlier results reached the agent (#20069) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Restarts must be invisible to a task: an agent that was working keeps working after the control plane comes back > - Restart chaos t03 (Daytona warm wave, K4 holds) ended a task as blocked even though its document was saved > - The cause is in how a terminal decision (paperclip_block or paperclip_finish) is handled after a long controller outage, not in how the Runner journals tool calls > - This pull request refuses a terminal call that the agent made before it had seen results that already returned, so the agent continues with those results > - The benefit is that a controller outage no longer turns into a false "blocked" task ## Linked Issues or Issue Description Restart chaos t03, wave `w-20261108T142906-s9501` (Daytona, codex, A1, three back-to-back K4 holds of 136 s, 252 s and 152 s). Refs #19831 (durable ownership plan). **What happened** The kept instance's run log shows the order of events: 1. The turn is accepted at 14:29:57. The runner then delivers nothing to the controller until 14:39:27. The holds keep the controller down, and the Runner keeps Codex's `item/tool/call` requests until it is reattached. 2. During that outage Codex keeps running. Its `write_document` call stays pending. The model is sampled again about once a minute and says "The save request is also waiting on Paperclip". It then calls `paperclip_block`, with the reason `tool_unavailable`. 3. After the reattach the Runner delivers the backlog in order. `write_document` runs and returns (the document is created: sourceSeq 152). Then the stale block arrives (sourceSeq 247) before Codex has acknowledged the write result: there is no `tool.execution.completed` for it before the block. The task ends blocked. **Codex behavior, checked directly** against the pinned Codex 0.160 app-server. A local probe sent one dynamic tool call and left it unanswered: - Codex does not time out. It re-samples the model at about 60 s and 127 s ("The save call is still waiting to return"). - A reply sent 130 s later is accepted (`durationMs` 130003), and the model finishes ("DONE"). So the reply is not lost. The agent's terminal decision was made without it. **Expected behavior** - A terminal call decided while earlier Paperclip results were still unseen is refused with a retryable error. - The agent continues with those results. In t03 it would see that the document was saved, and finish. ## What Changed - `packages/paperclip-runner/src/live/terminal-call-freshness.ts`: `unseenReturnedToolCalls` reads the turn's committed run events up to the terminal call. It returns each dynamic call that has a returned `tool_result` but no provider `tool.execution.completed`. `staleTerminalCallResult` builds the retryable `terminal_call_before_results` error, which names those calls. - `runnerd-codex-transport.ts` `#handleSemanticToolInput`: when such calls exist, `paperclip_finish` and `paperclip_block` get that error instead of being dispatched. The recovery path after a restart uses the same handler. A long-running call with no result yet does not count. - Crash point `runner.semantic_tool.after_input_before_dispatch`, just before the controller dispatches a dynamic tool call. A crash-point hold (`--hold-ms`, `--crash-point-hold-ms`; a K4-style receipt) keeps the controller down after the self-kill. Together they reproduce t03 on Daytona. ## Verification - Red/green: `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "refuses a terminal call decided"`. - The case `seen: false` failed on master: the handler was called and the stale block was dispatched. It passes with the fix. - The case `seen: true` (the agent saw the result first) dispatches normally, both before and after. - `npx vitest run src/live/runnerd-codex-transport.test.ts`: 289 passed. `src/live/terminal-call-freshness.test.ts`: 5 passed. - `npx tsc --noEmit -p packages/paperclip-runner` passes, and `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Shared registry: 15 passed. Call-site test: 3 passed. Chaos runner tests: 12 passed. - Codex probe (local, about 2.5 min per run): reproduced the "still waiting" re-sampling and the accepted late reply described above. - Not yet run: the Daytona crash-point repro with a hold. It needs the shared harness slot, and I'll add the result here. The fix is controller TypeScript, which is not an input of the Daytona image, so the current image still matches. ## Risks - A refused terminal call costs the agent one more turn step. It applies only when a result returned that the agent had not seen, which in practice happens only after a controller outage. - The check reads committed run events. If the store evicted the earlier call's events, it finds nothing and keeps today's behavior. - Upgrade-safe run state: no persisted shape changes. The check reads events the Runner already commits. A run saved by the previous release resumes with the same history and the same answer. - ACPX and other providers emit no dynamic `tool.execution.*` events, so the check is a no-op for them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 10:43:21 -06:00
DottaandClaude Opus 5.5	0746d1203d	feat(recovery): let the board confirm that an unrecorded owner stopped (#20067) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, recovery releases an in-progress record only once its owner is proven gone, never by age > - A run lease saved before owners were stamped has no owner Paperclip can judge on an ephemeral (multi-host) server, so it stays stuck and holds its run's sandbox forever > - Only a person can tell that the previous server stopped (for example, its container was replaced) > - This pull request asks the board on the stuck task and lets an owner confirm the stop, with an activity-log entry; recovery then releases the record on its normal path > - The benefit is that legacy stuck records have a way out that never guesses ## Linked Issues or Issue Description Follow-up to the Phase 2 durable-ownership work: #20054 (controller instance lease) and #19883 (process-space recording). It generalizes the existing "confirm the previous server stopped" action for native controllers (`native_controller_host_unverified`). No duplicate PR was found. **What happened?** On an ephemeral server, an active run lease of a finished run that was saved before holders were stamped has no owner stamp. The ownership reconciler reports it `unprovable (no_owner_stamp)` and never settles it. No operator action exists, so the lease and its sandbox stay reserved. **Expected behavior** The task shows the stuck resource, and a board owner can confirm that the previous server stopped. Recovery then releases the lease. Agents and operators without the owner-stop permission cannot. **Steps to reproduce** 1. Run with `PAPERCLIP_EPHEMERAL_SERVER=true`. 2. Have an active environment lease with no `leaseController` stamp, whose run is terminal. 3. Run an ownership reconciliation pass. The lease stays `unprovable` with no action on the task. **Paperclip version or commit** `master` before this PR. ## What Changed - `packages/shared`: `confirmOwnerGoneSchema` (`actionId`, `recordId`, `stopEvidence` of 20 to 12000 characters). - `recovery/owner-gone-attestation.ts`: - `requestOwnerGoneConfirmation(kind)` is an `onUnprovable` hook. For a legacy unprovable owner (`no_owner_stamp`, `process_space_unrecorded`, `missing_identity`) on a record that has a task, it opens one idempotent board recovery action on that task (cause `ownership_owner_unprovable`). - `confirmOwnerGone(...)` verifies the exact action and record. It asks the kind to write the attestation (compare-and-swap), resolves the action, and writes an `issue.ownership_owner_gone_confirmed` activity-log entry with the actor and the stated evidence. - Ownership registry: - `OwnableKind.attestOwnerGone` is a new optional hook. - `environment_run_lease_active` implements it: only an unstamped active lease of a terminal run, with the notification for the cleanup path. - The lease's owner then reads `operator_attested`, which proves gone, so the existing settle releases it. A holder stamped since the board was asked owns the lease, and its stamp decides. - Route: `POST /api/issues/<built-in function id>/recovery-actions/confirm-owner-gone`. It uses `assertBoard` plus `runtime:confirm_owner_stop`, the same board-only permission as confirm-controller-stopped. It is registered in OpenAPI. - UI: `issuesApi.confirmOwnerGone`. The task's recovery card shows the existing `PreviousServerStopConfirmation` for this cause, with resource-specific wording; owners confirm and operators see the wait. `isPreviousServerStopCause` includes the cause, so the generic card does not offer other resolutions. ## Verification - `owner-gone-attestation.test.ts`: - the stuck lease opens one board action on the task; - confirmation resolves the action, writes the activity log, and the next pass settles the lease (`operator_attested`) exactly once; - a mismatched record is refused with 409; - a holder stamped after the ask is refused and the lease stays active; - no action for a judgeable owner or for a record with no task. - Red, with the `onUnprovable` hook removed: 2 of 3 fail. - UI: `WorkspaceExportRecovery.control.test.tsx` checks that the owner confirms with the right payload and that an operator sees the wait without the control (47/47). - Server suites: openapi-routes, ownership and recovery suites, plus the existing native-controller-host recovery. Low-memory server typecheck is clean; `check:token-gates` is clean. ## Risks - The board can attest a server stopped that is in fact running. The action and the UI copy ask for the evidence, and the activity log records who attested what. This is the same trust model as confirm-controller-stopped. - Only run leases opt in so far. Records without a task (login/probe leases) keep the legacy idle sweep. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 09:46:36 -06:00
DottaandClaude Opus 5.5	66a2cbab98	fix(runner): adopt the in-flight Codex turn after a restart lands just after turn.start (#20057) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run survives a server restart by resuming its durable provider session and adopting the turn that was in flight > - The adoption rule (RC-59) only adopts a turn named by the latest `turn.start` command's accepted result > - Two gaps made it fail. A restart just after `turn.start` leaves that command `execution_indeterminate` (the runner died after journaling it). Separately, the rule demanded that Codex echo the requested turn id, which it never does. > - The restored session then rejected Codex's own in-flight turn as unexpected, and the user's Retry failed on the first click > - This pull request adopts exactly the turn the runner reconciled for that ambiguous start, and accepts Codex's own turn ids > - The benefit is that a restart landing right after a turn starts no longer costs the user a second Retry ## Linked Issues or Issue Description Found by the restart-chaos campaign: wave `w-20261108T134237-s9401`, task t77 (Codex C1), Retry run `5358f1cb`. Related: #19852 (fast follow-up turns), RC-59 adoption. No duplicate PR was found. **What happened?** The Retry run submitted its first turn at 14:05:25.171. A K2 SIGTERM followed at .690 (graceful drain, then `resume_dead_runner`). - The runner had journaled the `turn.start` and then died, so on restore it settled the command as `failed / execution_indeterminate`. - The runner's restored Codex session reconciled the ambiguous start to the live replacement turn (`session.reconciled`, `activeProviderTurnId`). - Codex then sent `turn/started` for that turn. The restored driver had no active or pending turn, because `controllerAcceptedTurnId()` was null, and failed with `turn_binding_mismatch`. - The run ended `native_event_replay_conflict`, and the user needed a second Retry. **Expected behavior** The restored session adopts the runner-reconciled in-flight turn and finishes it. It does not treat it as a replay conflict. **Steps to reproduce** 1. Start a native Codex run. 2. Send SIGTERM to the server about 0.5 s after the first `turn.start`, before runnerd confirms it. 3. Let the server restart and resume the run (`resume_dead_runner`). 4. Before this PR, the run fails with "Provider started an unexpected turn." **Paperclip version or commit** `master` before this PR. ## What Changed - `controllerAcceptedTurnStartId(commands, options)` (runnerd-codex-transport.ts): - **Codex and OpenCode results.** These providers assign their own turn ids, so an accepted result never echoes the requested `turn_lab_` id, and RC-59 adoption never matched a real Codex result. For them, the accepted turn is now adopted only while it is also the runner's live active turn in its authenticated `session.snapshot`. A turn the restored runner reports lost has no live turn, so it keeps the established restart-continuation path. ACPX still requires the echoed id. - **Indeterminate start.** When the latest `turn.start` (after the latest attach) failed with `execution_indeterminate`, it adopts the runner's live active turn. Runnerd reports one only after it reconciles an ambiguous start to a distinct replacement turn. - The transport records the snapshot's active turn at `thread/read`, which `recover()` reads before it evaluates adoption. - The driver's existing RC-59 guards still apply: the turn must be in provider history, be the only active turn, and not already be terminal. ## Verification - Connected restart test, `runnerd-restart-continuation.test.ts` "adopts the in-flight turn when a restart lands just after turn.start (t77)". It uses the real runnerd and fake Codex: - the runner and provider die after journaling a started turn; - runnerd's own restore settles the command `execution_indeterminate`; - recovery adopts `provider-turn-1` with no second `turn/start`. - It fails without the fix: nothing is adopted. - Unit cases for `controllerAcceptedTurnStartId`: - a Codex-assigned id is adopted only while it is the live turn; - an indeterminate start is adopted with the reconciled turn; - it fails closed with no live turn, an ordinary failed start, a later attachment, or an ACPX id mismatch. - Regression guarded: "starts a new turn when the restored runner reports a lost turn the checkpoint never bound" still passes. - `runnerd-codex-transport`, `codex-app-server-driver.recovery`, `runnerd-codex-transport-adoption`, `runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation` and `runnerd-controller-outage`: 383/383. Runner TypeScript typecheck is clean. - Live: the chaos campaign will run the `runner.turn_start.*` crash points (Codex A1/C1) and a Codex C1 K2 storm, and post the result here. ## Risks - **Previous-release story.** No persisted shape changes. Commands journaled by the previous release carry forward and are read with the new rule. An indeterminate start that the runner did not reconcile to an active turn still fails closed, as before. - **Wider Codex adoption.** Codex adoption now actually fires. If runnerd ever returned an accepted result that names a turn other than the one this command started, the driver would adopt it. Runnerd binds the result to the exact command id, so this is the same trust the rule already placed in ACPX. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 09:26:58 -06:00
Nicky Leach	17ad23aafe	fix(workspaces): back off failed terminal cleanup and retire abandoned reservations through remove (#20020)	2026-11-08 07:12:48 -08:00
DottaandClaude Opus 5.5	96badeaa42	fix(lifecycle): never repeat a passed harness verification; settle a dead verifier's lease at once (#20030) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A new or reconfigured agent runs a harness verification (a hello probe in a test sandbox) before it is ready, and its runs wait until it passes > - In the final Daytona restart chaos wave, a Codex agent's runs waited about 417 s for their first sandbox, while boot took only about 32 s > - Each control-plane restart killed the verifier. The next process waited out the 120 s lifecycle lease and then booted a fresh sandbox. Nothing recorded a configuration that had already passed. > - This pull request records a passed verification by what makes it valid and reuses it, and settles a dead verifier's lease on the shared ownership path at once > - The benefit is that a restart never repeats a passed verification, and never leaves runs waiting out a dead process's lease ## Linked Issues or Issue Description **What happened?** In the final Daytona chaos wave, the Codex agent was created at 09:55:46 UTC and its three runs were queued. Verification ran four times, in four sandboxes, because faults K1, K4 and K2 each killed the verifying server. After K2 the server was ready at 10:00:20, but verification restarted only at 10:02:13: it waited out the 120 s lifecycle lease. The first run sandboxes were created at 10:02:43, 417 s after launch. **Steps to reproduce** 1. Create a Codex agent and assign it a task at once, so the task waits for harness verification. 2. Kill the control plane (SIGKILL) while the agent is verifying. 3. The restarted server waits out the dead verifier's 120 s lifecycle lease, then verifies again in a new test target. The task's first token arrives about 130 s after the fault. **Expected behavior** - A passed verification of the same effective configuration in the same runner image is never repeated. - A verification is never restarted from scratch while its holder process is alive. When the holder is proven gone, its lease is settled at once. - Runs gate only on a configuration that has never passed. **Root cause** `verify()` (`server/src/services/agent-harness-verification.ts`) leased a fresh sandbox on every attempt and recorded nothing on a pass. The lifecycle lease (`modules/agent-lifecycle/adapters/postgres.ts`) named no holder process, so a restart could only wait for it to expire. ## What Changed - Restart chaos TTFT driver: a `verify` phase, reported in its own rows, and a `--verify-fault K1|K2|K6` option. The fault fires 1 s after `verifying` is observed, so it interrupts a held lease. The seed keeps each agent's creation body, so the driver can hire an identical agent. - **Persisted pass:** new columns `agents.harness_verified_digest` and `harness_verified_at` (migration 0448). The digest is an HMAC, keyed with the instance's decision-signing secret, over: - the fully resolved effective configuration: every value the test used, including `cwd`, resolved credentials, and agent-over-environment precedence; - the selected AI account and its credential generation (connection and grant ids and `updatedAt`, never the credential); - the environment and the runner image. The key means the stored value cannot confirm a guessed credential offline. For a sandbox, the runner image is an immutable image digest or a qualified Daytona snapshot ID (`sandboxBinding`, now exported). For local runs it is the server version. A mutable image or snapshot name, or a local environment redirected to the managed sandbox, never reuses a pass. - **Server-only:** `normalizeAgentBaseRow` and `redactAgentRowForResponse` strip the digest and its timestamp from agent API responses. - `verify()` still runs every account, policy and secret check, then returns complete on a matching digest without booting a test target. It records the digest on a pass before the lifecycle step completes, so a restart in between reuses it. - **Lease holder:** the lifecycle `claim()` stamps the claiming controller process on the lease (`lifecycleOperation.leaseHolder`, supplied through the lifecycle effects port) and clears the stamp with the lease. The compare-and-swap release lives in the lifecycle module (`releaseAgentLifecycleLeaseOfHolderInTransaction`). - **Shared ownership path:** a new `agent_lifecycle_lease` kind with `settleAgentLifecycleLease`. - When the reconciler proves the holder gone, settle clears exactly that lease (compare-and-swap on the holder), notifies lifecycle work, and logs `agent.lifecycle_lease_reconciled`. - A live holder is never taken over. The lifecycle state, operation and progress are untouched. ## Verification - `restart-verification.test.ts` (embedded Postgres, real lifecycle store and worker, real reconciler pass): - A verification held by a dead process is claimed and completed at once after the startup pass, with the agent ready in under 10 s instead of waiting out the 120 s lease. It fails without the new kind. - A live holder is never taken over. - The claim stamps the holder, and `defer` clears it. - `agent-harness-verification-preflight.test.ts`: - A second `verify()` of the same configuration allocates no target. A revoked account still fails before reuse. A changed model, `cwd` or literal credential tests again. It fails without the reuse. - A failed verification records nothing. - Ownership report, all lifecycle module tests, remote, provider-API and unit harness verification tests, and crash-point call sites: 13 files, 220 tests pass. - Targeted local run: the TTFT driver's new `verify` phase (`--phases verify --verify-fault K1`, Codex, 2 samples) hires a new agent, assigns it a task at once, kills the control plane (K1) while it verifies, and times fault to first agent token: - Before (master server code): 129.8 s and 129.8 s. The restarted server waited out the dead verifier's 120 s lease. Run-log TTFT from run creation was only 4.1 s. - After (this branch): 15.8 s and 17.2 s. The startup reconciler pass settled `agent_lifecycle_lease` and verification re-ran at once. ## Risks - **A stale pass:** the digest covers every resolved value, the selected account generation, the environment and an immutable runner image, so any change tests again. Rotating the instance signing secret invalidates every digest, so each agent verifies once more. - **Migration:** two nullable columns, no backfill. Existing agents verify once more, then reuse. - **Settle:** it runs only on a `gone` proof, compare-and-swap on the exact holder. It never changes the lifecycle state. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 08:10:40 -06:00