3659 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	90404d2033	perf(run-logs): read object-store log tails in parallel, once, and cached (#20284) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances keep run logs in object storage as a journal of content-addressed segments and binary index nodes > - The task chat reads the last 32 KB of a run's log, aligned to a record, to show what the agent did > - A run that flushes often has many small segments. The tail read fetched each object one after another, and record alignment walked the index a second time > - On staging, one tail read took 11–12 s, and the task chat waited for it (see #20277) > - This pull request reads the tail in parallel, in one pass, and caches the objects that cannot change > - The benefit is that the first tail read is about 6.6x faster, and repeated reads (live polling, reopening a task) use only the mutable head ## Linked Issues or Issue Description **What happened?** `GET /api/heartbeat-runs/:runId/log?offset=…&limitBytes=32000&alignStartToRecord=true` took 11–12 s on staging for a 400 KB object-store log. Without alignment it took 6.1 s. A read from offset 0 took 2.0 s. The task chat waited for this read before it showed the run's output. **Expected behavior** A 32 KB tail read returns in well under a second, and a repeated read of the same tail costs about one storage round trip. **Steps to reproduce** 1. Use object-backed run logs (`s3_segments`). 2. Write a run log in many small flushes, as a live run does. 3. Read its last 32 KB with `alignStartToRecord=true`, and count the object GETs and how many run at the same time. **Paperclip version or commit** `master` at 6459488984. **Deployment mode** Hosted (authenticated) deployment with S3-compatible run-log storage. Refs #20276 (measurement and plan), #20277 (chat no longer waits on a slow transcript). ## What Changed - `object-byte-journal.ts`, `readRange`: - Visits the index tree and fetches sibling objects concurrently, with at most 4 fetches for each read. - Reassembles the pieces by offset. - Cancels the remaining fetches when one fails. - The shared `objectIO` limiter still caps the process at 8 slots, and background persistence still gets freed slots first (#19176). - `object-byte-journal.ts`, `load`: - Keeps segments and index nodes in a process-wide LRU, limited to 32 MB in total and 1 MB per object. The budget counts each entry's key and a fixed 256-byte overhead with its bytes. - These objects cannot change: each key ends in the SHA-256 of its bytes, and `load` verifies that digest before it caches the object. - The mutable head is never cached. - The cache and the shared fetches are scoped to the storage provider instance. Production uses one provider in the run-log store singleton. - Concurrent fetches of the same object are shared. A shared fetch has its own controller. A reader that aborts only stops its own wait. The fetch is aborted only when no reader still waits for it, so the other readers never start a second GET for that object. - `object-run-log-store.ts`, `read`: - With `alignStartToRecord`, the first 4 KB alignment window is read in the same range as the body. - The existing 32 KB backward windows still handle longer records. - The output is unchanged. - New test `object-run-log-store-tail-read.test.ts`, written against a fake provider that counts GETs: - correct aligned and unaligned content - concurrent fetches - alignment costs at most 30 extra GETs, with no second walk of the index - a repeated read costs 1 GET (the head) - appended output is visible after cached reads - a reader that shares a fetch survives another reader's abort - a reader that joined a shared fetch stops at once on its own abort - when the reader that started a fetch aborts, no object is fetched twice - a shared fetch stops when its last reader aborts - one store's cached objects are not served to a reader of another store - a tail that starts inside a record longer than the 4 KB lookback (6 KB and 90 KB records) - the cache budget counts per-entry overhead - the shared-fetch tests hold object GETs open with a gate instead of wall-clock delays ## Verification - `cd server && npx vitest run src/services/object-run-log-store-tail-read.test.ts src/services/object-run-log-store.test.ts src/services/object-byte-journal.test.ts src/services/run-log-store.test.ts`: 4 files, 74 tests pass. The tail-read file passed 5 of 5 repeated runs. - The 18 server test files that use the journal, the run-log store or `readLog`: 381 tests pass. Two skill-materialization tests timed out under load and pass when run alone (4/4). - Red checks: - On `master`, the aligned tail read made 357 GETs with at most 1 running at a time, and a repeated read made 356 GETs. - When I removed the abort handling for shared fetches, the shared-abort test failed 3 times out of 3. - The tests for the review fixes (joiner abort, no duplicate GET after the first reader aborts, last-reader abort, per-store cache scope, cache entry overhead) fail without the fix. - An off-by-one in the long-record lookback arithmetic makes both long-record tests fail. - Benchmark: same fixture, with 30 ms latency added to each GET, a 32 KB tail of 1,200 single-record segments. | Read | `master` | This PR | |---|---|---| | Aligned | 11,753 ms, 365 GETs, 1 at a time | 1,778 ms, 205 GETs, 4 at a time | | Unaligned | 5,979 ms, 184 GETs | 1,607 ms, 184 GETs | | Repeat aligned | 11,686 ms, 365 GETs | 34 ms, 1 GET | - `pnpm typecheck` in `server` passes. ## Risks - **Memory:** the cache can hold up to 32 MB of decoded log bytes in each server process. Authorization still happens in the routes before any read. The cache key is the full object key, which includes the run's journal prefix. - **Storage concurrency:** one read now holds up to 4 I/O slots instead of 1. Background persistence still gets freed slots first. Under many simultaneous cold reads, interactive reads can reach the 32-waiter limit sooner and receive the existing retryable 503 `object_journal_io_busy`. The cache makes repeated reads cheap, which offsets this. - **Remaining cost:** the first read of a cold tail still walks a binary index over small segments, about 1.8 s at 30 ms per GET. A larger gain needs writer-side changes, such as wider index nodes or compaction when the log closes. Those change the persisted shape, so they are not in this PR. - No migration, configuration change, persisted-shape change or new dependency. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:26:36 -06:00
DottaandClaude Opus 5.5	099adb2728	fix(recovery): settle a remote workspace save that stopped without finishing (#20291) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in remote sandboxes save the workspace as snapshots. Each save is a `workspace_operations` row with phase `remote_workspace_snapshot` > - The ownership registry lists that kind as in-progress work, but it had no settle step > - When a restart cut off a failing final save, the row stayed `running` after its invocations had all stopped and its run had ended > - The retained-source fallback refuses while any workspace operation is running, so a task whose saved files could not be verified could never continue from its last save > - This pull request lets the ownership reconciler settle that row as failed > - The benefit is that the automatic fallback can continue such a task without any user action ## Linked Issues or Issue Description **What happened?** A native run in a remote sandbox finished its turn. Its final workspace save failed because the sandbox disk was full ("No space left on device" in the snapshot copy). A graceful restart then interrupted the retry. The run ended as `native_workspace_sync_out_retry_exhausted`. Its `remote_workspace_snapshot` row kept `status: running` for six days, with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }` and `exportCleanup: { state: "pending" }`. The retained-source fallback checked the task and refused with `workspace_operation_running`, so the task stayed blocked. **Expected behavior** A save with no active or unknown invocation, whose run is terminal, is finished work. The reconciler must mark it failed, so the fallback can continue from the last committed save. A save that might still be running must stay untouched. **Steps to reproduce** 1. Run a native task in a remote sandbox. Make its final save fail, then restart the server during the retry, so the save's work is quiesced and the operation stays `running`. 2. Let the task's recovery refuse with an unverifiable-source failure (for example `allocation_ownership_ambiguous`). 3. The retained-source fallback refuses with `workspace_operation_running` on every check, and the task never continues. **Paperclip version or commit** `master` at f7c85bca18. Refs #20282 (the other cause of the same stuck task: a false ownership refusal). ## What Changed - `settleAbandonedRemoteSnapshotSave` (`recovery/workspace-finalize-attempt-settlement.ts`) settles a `remote_workspace_snapshot` row in `running` or `pending` only when all of these are true: - its save work is quiesced (`activeIds` is empty and `unknown` is false); - its run is terminal. It sets `status: failed`, records `abandonedSave` provenance and `lastFailure: remote_workspace_save_abandoned`, keeps `saveWork` and `exportCleanup`, writes a `workspace_operation.reconciled` activity entry, and wakes the workspace's terminal work. - The ownership registry wires it as the settle of `remote_snapshot_save`. - In the same transaction it re-arms blocked fallbacks. For every open hold whose fallback check last refused with `workspace_operation_running` and whose source run used this save's workspace, it clears the check and wakes the workspace-recovery queue. A hold that had already spent its 12 checks (no next check) continues as well. - The settle uses the save path's own `readSaveWork`, now exported, so it never settles save work that the save path would treat as invalid. - Because the row now has a `lastFailure`, `retireQuiescedSaves` retires it like any failed save when a later final save commits. A later retry of the same save boundary reopens a failed row, as before. - The ownership inventory row for this kind names the new settle. ## Verification - Red before the fix, green after: `settles a remote snapshot save that stopped without finishing after its run ended, and keeps open or live ones` (`ownership-reconciliation-report.test.ts`). Before: `settled: 0`. After: only the abandoned save settles. Saves with a live run, an active invocation, or unknown save work stay `running`. - Red before the fix, green after, end to end: `continues once the reconciler settles a final save a restart left running after its run ended` (`workspace-recovery-fallback.test.ts`). The fallback first refuses with `workspace_operation_running`, and the test then spends its check budget (12 checks, no next check), so the hold drops out of discovery. After one ownership reconciliation pass, the save is `failed` and the check is cleared. The worker then resolves the hold with `retained_source_unverifiable_continued_from_last_saved`. Without the re-arm, this test fails. - The reconciler test also covers save work that is not quiesced (`quiescedAt: null`), save work that is invalid (`quiescedAt: ""`), and a `pending` save. - `workspace-recovery-fallback.test.ts` (16 tests), `ownership-reconciliation-report.test.ts`, and the remote workspace lifecycle, save-concurrency, and snapshot-retention suites pass (80 tests, 1 environment-gated file skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed. ## Risks - A save in another process registers its invocation in `activeIds` in the same row-locked transaction before it runs. The settle locks the row and requires `activeIds` to be empty and `unknown` to be false, so it cannot fail a save that is still working. - A save whose process died with its invocation still in `activeIds` stays `running`. That case needs an owner-death proof, which is out of scope here. - No schema or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only browser diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:26:04 -06:00
DottaandClaude Opus 5.5	ef2b8d5324	fix(issues): make every Retry start work or say why (#20283) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a task's run fails, the board recovers it with a Retry button, so that button must either start work or say why it did not > - An audit of every Retry-type button found three gaps. The task Retry shown during a scheduled retry's countdown returned "accepted" but changed nothing. A stale Retry looked like success in the Inbox. "Retry now" skipped the task's read and change checks, so a member who cannot see a private task could start its retry > - This pull request makes the task Retry start a scheduled retry immediately, makes every Retry that did not start work say why, and gives "Retry now" the same authorization as the task Retry > - The benefit is that no Retry click silently does nothing ## Linked Issues or Issue Description Refs #19831. The board reported a dead Retry button. A read-only audit of every Retry button confirmed: - **Task Retry during a scheduled retry was a no-op.** `TaskInterruptionPresentation.canRetry` is true in `retry_scheduled`, so `ExecutionBlockerNotice` shows Retry. `requestTaskRetry` treats `scheduled_retry` as busy and inserts a `coalesced` wake pointing at that run. `resumeTaskRetry` reads only `deferred_issue_execution`, and dispatch uses coalesced user wakes only to carry comments. Nothing promoted the run: the API answered 202 `accepted: true` and the countdown did not change. - **A stale Retry looked like success.** `accepted: false, stale: true` comes back as HTTP 200. The Inbox ignored `accepted` and opened the task as if work had started. The task page showed no reason. - **"Retry now" checked less than the task Retry.** It ran only `assertBoard` and `getAccessibleResource`, skipping `assertIssueReadAllowed` and `issue:mutate`. A company member who cannot read a private task could promote that task's scheduled retry. - **"Retry now" said nothing for some outcomes.** It toasted only `promoted` and `gate_suppressed`. ## What Changed - `server/src/services/task-retry.ts`: when the Retry coalesces into the task's latest run and that run is `scheduled_retry` (with no other Retry already active), the receipt returns `scheduledRetryRunId`. - `server/src/routes/issues.ts`: - `POST /issues/:id/retry` then promotes that run through the existing `retryScheduledRetryNow`, the same path "Retry now" uses. It logs `issue.scheduled_retry_retry_now` with `source: "task_retry"` and returns `scheduledRetry: { outcome, message }`. - `POST /issues/:id/scheduled-retry/retry-now` now runs `assertIssueReadAllowed` and `decideIssueAccess("issue:mutate")`, matching `/retry`. A private task the member cannot read now answers 404. - `ui/src/hooks/useTaskRetry.ts`: new `taskRetryNotStartedReason`. When a Retry did not start work (stale, or a scheduled retry that is gate-suppressed, waiting, or gone), the hook returns that reason as its error. `ExecutionBlockerNotice` and `StalledDependencyNotice` already display it. - `ui/src/pages/Inbox.tsx`: a task Retry that did not start work shows the reason in the error toast and does not navigate. - `ui/src/hooks/useRetryNowMutation.ts` and `IssueScheduledRetryCard.tsx`: every "Retry now" outcome now shows its headline and message, including `no_scheduled_retry` on the card. - `ui/src/api/issues.ts`: `retryTask`'s response type includes `scheduledRetry`. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-scheduled-retry-routes.test.ts`: - "starts a scheduled retry now when the task Retry is clicked during its countdown": 202 with `scheduledRetry.outcome: "promoted"`, the same run becomes `queued`, and no second retry run is created. - "refuses Retry now on a private task the member cannot see, like the task Retry": on master "Retry now" answers 200 `promoted`; now both routes answer 404 and the run stays `scheduled_retry`. - The seed now adds the board user's membership, which the stricter route needs. All 33 tests pass. - `ui/src/hooks/useTaskRetry.test.tsx`: a replaced attempt, a gate-held scheduled retry and a waiting scheduled retry each show their reason; a promoted one shows nothing extra. - `ui/src/pages/Inbox.test.tsx`: an Inbox Retry that did not start work (stale, or gate-held) toasts the reason and does not navigate. - `ui/src/components/IssueScheduledRetryCard.test.tsx`: "explains when there was no scheduled retry left to start". Other Retry buttons: the chat failed-run Retry already reports stopped and failed retries (`agents.test.ts`). Agent lifecycle "Retry now" and cost "Retry accounting" start no task work and already surface their API errors. Regression and static checks: - UI: 7 files, 168 passed. Server: task-retry and scheduled-retry suites, 149 passed. `issue-agent-mutation-ownership-routes.test.ts` passed. - `maintenance-server-routes.test.ts` "assembled maintenance server production wiring" fails identically on master on this Mac (`ensure-build-deps`). - `tsc --noEmit` passes for `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A task Retry during a scheduled retry now starts the run immediately, through the existing "Retry now" promotion gates (pause, budget, cleanup waits). A gate that holds it is reported, not bypassed. - "Retry now" is stricter: a member without read or change access to the task gets 404 or 403, as on `/retry`. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:25:21 -06:00
Dotta	5c73633a8f	fix(workspaces): preserve remote recovery holds during online import (#20298) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace import preserves old controller-local files while unrelated remote work continues. > - A task can have an old local checkout and a separate durable remote workspace. > - A terminal remote run can retain a file-recovery hold on that task. > - The legacy source checker rejects remote references, so that hold blocks independent local imports. > - This pull request proves that the hold belongs only to the separate remote source and fences its finalizer during import. > - Old files can migrate while the remote recovery obligation stays intact. ## Linked Issues or Issue Description Refs #19873 and #19854 for the existing cohort and source-verification paths. Searched related online migration and remote recovery PRs; no duplicate fix was found. **What happened?** Online import rejects a legacy Git checkout with `online_workspace_source_scope_unproven` when a same-task failed run has a prepared reference to a separate durable remote workspace. The legacy run reader rejects remote references before the hold can be classified. **Expected behavior** Independent local files can import under their source fences. The separate remote workspace, dirty writer, allocation, accepted result and recovery hold must stay intact. **Steps to reproduce** 1. Create related local Git workspaces. 2. Record a terminal failed native run and an exhausted file-recovery hold for a separate durable remote workspace on the same task. 3. Import the local leaf online. The old checker refuses before capture. **Paperclip version or commit** Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`. **Deployment mode** Managed server with PostgreSQL and remote Runner execution. ## What Changed - Add a distinct remote-only preservation proof. Keep remote runs excluded from legacy export sources. - Use the canonical source inspector and prepared-reference matcher. Bind the run, finalizer, action, task, accepted result, authority, allocation and prior committed snapshot. - Acquire the held run's finalizer fence through the existing sorted lock path. Existing source-proof reinspection rejects changes before publication. - Preserve previous-release prepared references with an older baseline snapshot. Do not require that snapshot's epoch or source run to equal the current run. - Add real PostgreSQL import, authenticated cold restore, preservation, refusal, lock-contention and publication-race regressions. Document the behavior. ## Verification - Genuine baseline failure on unchanged product: the real importer refuses with `online_workspace_source_scope_unproven`. - Final focused regression file: 23 tests passed in 13.19 seconds through the canonical isolated PostgreSQL wrapper on Node 24.22.2. - 24 existing affected migration/recovery tests passed against the same final production bytes. Another 83 tests were explicitly excluded by the focused selection; no full-suite pass is claimed. - The positive imports local bytes, confirms the remote finalizer fence is held during upload, leaves all remote records unchanged, parks the original repository and authenticates/restores the archive. - Refusal and race cases cover malformed, foreign, same-local and mixed references; active owners/writers; changed allocation, result, task, action, authority and baseline; and finalizer contention. - Server TypeScript check, module-boundary check and `git diff --check` passed. - An intermediate run had one incorrect assertion that expected an explicit empty legacy-source array. The canonical journal omits that empty field. The final assertion accepts absence and still forbids any exported remote source. Final relevant results are 23 new plus 24 existing passes, rather than a claim that the earlier invocation passed in full. - Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`: 55 successful checks and 4 intentional skips. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42650025033). Greptile reviewed that head at 5/5 with no actionable findings or open inline/review comments. Hosted continuation remains pending normal delivery. ## Risks - A false disjointness proof could admit an unsafe local import. Strict company/task/source binding, the canonical inspector, finalizer fencing and publication revalidation retain refusal on uncertainty. - This does not recover or settle remote files. Dirty authority and the original recovery obligation remain unchanged. - No schema, public API, archive format or saved-run identity changes. Existing persisted state carries forward. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 10:13:41 -06:00
DottaandClaude Opus 5.5	d10c6e0e8b	fix(chat): hold a chat message at the agent's daily limit instead of dropping it (#20292) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat sends each person's message to the conversation agent through a durable outbox (`deliverConversationComments`). The outbox keys each wake receipt as `conversation-comment:<commentId>`. > - The outbox treats any receipt with that key as delivered. Queue admission writes a `skipped` receipt when the agent reached its daily run or cost limit. > - So a chat message sent after the agent used its daily allowance was consumed silently. No run started, the chat showed nothing, and nothing resumed it. > - A person's message must never be silently dropped. The daily limit is an operator control, so the message must wait honestly, not bypass the limit. > - This pull request makes the outbox hold the message at the daily limit, like it already does for a paused agent, and resume it when the limit resets. > - The benefit is that a chat message at the daily limit is saved, shown with its reason, and sent automatically at 00:00 UTC. ## Linked Issues or Issue Description No public issue exists. Inline bug report: **What happened** In a chat eval, the conversation agent (CEO) had `maxDailyRuns: 3`. After three chat turns and a separate failed-model phase on another agent, the person sent a fourth message. The comment was saved (POST 201), but no CEO run started. The conversation stayed `active` with no `executionRunId`. The ownership reconciler later reported `I7_visible_next_action`. The server log shows no wake error. Queue admission had written a `skipped` receipt (`heartbeat.daily_run_limit`) under the outbox key, so the outbox counted the message as delivered. **Expected behavior** A chat message that reaches the agent's daily limit is not consumed. It stays saved, the chat shows why it waits, and it is sent automatically when the limit resets. The daily limit still applies: the message does not start a run early. **Steps to reproduce** 1. Set a conversation agent's `runtimeConfig.heartbeat.maxDailyRuns` to N. 2. Send N chat messages and let each turn finish. 3. Send one more chat message. 4. See that no run starts, the chat shows no waiting state, and the message is never answered (a `skipped` wake receipt with reason `heartbeat.daily_run_limit` exists for `conversation-comment:<id>`). ## What Changed - `server/src/services/agent-conversations.ts`: add `conversationDeliveryHold`, which applies the same daily run/cost limit check as queue admission. The outbox returns `deferred` at the limit and writes no receipt, so the message stays owed. It reports the next UTC day start through a new optional `onDeferredDeadline` option. - `server/src/services/recovery/service.ts`: stranded-work recovery passes its `onDeferredDeadline` to the outbox, so the held message is retried when the limit resets. - `server/src/services/heartbeat/queue.ts`: if the limit is reached while an outbox wake is in admission (a race), admission throws a 409 with `reason: heartbeat.daily_*` for `conversation-comment:` keys instead of writing a `skipped` receipt. The outbox already maps a matching 409 to `deferred`. Other wakes keep the current skip behavior. - `server/src/routes/issues.ts` (`GET /issues/:id/queued-comments`): when a chat has no queue receipt but has messages held by the daily limit, return them as a `deferred` queue with `executionWait` (reason and message). Edit, send and discard stay off because the outbox owns delivery. - `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: show the `executionWait` message for such an outbox-held queue (no queue id and no target run). Receipt-backed queues keep the current behavior and do not duplicate the task status. - Tests: three real-database tests in `agent-conversation-delivery.test.ts` (hold and resume after the limit resets, recovery deadline, and a limit reached during real `heartbeatService.wakeup` admission) and one UI test. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversation-delivery.test.ts`: 10 passed. Without the source change, the three new tests fail (`expected 'delivered' to be 'deferred'`; the real queue path writes the skipped receipt). - `cd server && pnpm exec vitest run src/__tests__/issue-recovery-conversations.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-undispatched-execution-wait.test.ts src/services/heartbeat/queue.test.ts src/__tests__/agent-chat-messages.test.ts`: 215 passed. - `cd ui && pnpm exec vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx`: 36 passed. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: clean. `node scripts/check-token-gates.mjs`: clean. - Manual: give a chat agent `maxDailyRuns: 1`, send two messages. The second message shows in the queued messages bar with "<agent> reached its daily run limit. Your message is saved and sends automatically when the limit resets at 00:00 UTC." ## Risks - Behavior change for chat only: a chat message at the daily limit now waits in the outbox instead of being consumed. It is sent after 00:00 UTC. If an operator raises the limit, the message is sent on the next recovery pass for that chat; the UI message does not promise an earlier send. - Saved state from the previous release: no persisted shape changes. Messages that an earlier release already consumed with a `skipped` receipt stay consumed. They are not replayed later, which avoids sending old messages without notice. - The queued-comments read adds one agent read and one daily-limit count for a chat with no queue receipt. This is the same query that admission runs. - The recovery deadline is in-process; instance-sleep scheduling does not get a new deadline from this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 10:03:28 -06:00
DottaandClaude Opus 5.5	87e755ce8e	fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in remote sandboxes keep a warm sandbox and hand it to the next run through a chain of reusable leases > - Acquisition binds that chain to the execution workspace, so a task can inherit a warm sandbox from an earlier task in the same workspace > - Saved-files recovery verifies the same chain, but it required every earlier lease, run, and finalization to belong to the recovering task > - A valid chain that started with another task therefore always failed with `allocation_ownership_ambiguous`. That code is `service_required`, so recovery stopped after one attempt, and each board Retry repeated the same refusal > - This pull request makes the chain check match the acquisition rule, and it checks each predecessor against its own task > - It also makes the board say plainly when recovery refuses the saved state, because "failed after automatic retries" was not true and Retry looked like it did nothing > - The benefit is that a task blocked with "Saving the agent's files failed after automatic retries" can recover its saved files, and a refusal that Retry cannot change is shown with its reason ## Linked Issues or Issue Description **What happened?** A native run in a remote sandbox finished its turn, and its result was accepted. The copy of its files back to the host failed and was scheduled to retry. A graceful server restart then interrupted the retry, so the run settled with `native_workspace_sync_out_retry_exhausted`, and the task became blocked. Every saved-files recovery generation then failed on its first attempt with `allocation_ownership_ambiguous` (`service_required`). The board showed "Saving the agent's files failed after automatic retries. Retry to try again". Each Retry replaced the generation (`issue.recovery_generation_replaced`), and the new generation failed in the same way within one second. To the user, Retry did nothing. The sandbox lease chain was valid: same agent, same execution workspace, same sandbox, and matching handoff receipts on all eight leases. But its first three leases belonged to an earlier task in the same workspace. **Expected behavior** Recovery must accept a warm sandbox chain that acquisition accepted. Acquisition binds reuse to the execution workspace, and it requires the same task only when the lease has no workspace. Recovery must check each predecessor run and finalization against that predecessor's own task. It must still refuse a chain whose records disagree or that has no shared workspace. **Steps to reproduce** 1. Run a native task in a remote sandbox with `reuse_by_environment`, with an execution workspace that another task also uses. 2. Run the other task in the same workspace first, so the sandbox lease is handed from that task to this one. 3. Make the last run of the second task fail to copy its files back (for example, restart the server during the copy-back retry). 4. Wait for saved-files recovery, or click Retry. Recovery fails with `allocation_ownership_ambiguous`, and Retry repeats the failure. **Paperclip version or commit** `master` at a3a25ce6c9 (the same code as the 2026.1106.0 staging build for every file this change touches). ## What Changed - `retiredReusableLeasePredecessors` (`environment-lease-handoff.ts`) requires the same task only when the lease has no execution workspace. This matches the binding that `acquireLease` uses when it replaces a reusable lease. - `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`) checks each predecessor run and its finalization against the predecessor lease's own task, not the recovering task. A predecessor from another task is accepted only when both leases have the same execution workspace. All other checks stay: same agent, terminal run, no live controller or lease owner, a settled finalization, and a saved snapshot at an older epoch on the same allocation, workspace, and remote directory. - The other callers of the chain walk (remote termination resolution and retired backup restore) get the same rule. - The test fixture `warmHandoffSource` gets an option that starts the chain with an earlier task in a shared workspace. - Honest copy for exhausted recovery (`exhaustedSystemRecoveryAction`): - A deterministic refusal (`service_required`) now says "Paperclip refused to recover the saved files", names the reason in plain words and the code, and says that Retry runs the same check again and succeeds only after Paperclip is updated. - Other exhausted recovery says "stopped after N attempts" only when there were several attempts. One attempt says "failed". - Continuation (`explicit-native-continuation.ts`) reports a new wait reason, `saved_work_recovery_refused`, when the exhausted recovery is a deterministic refusal. The task presentation shows its own message for it. `saved_work_recovery_failed` no longer claims automatic retries. - Retry stays available for a refusal. It is the action that runs recovery again after an update. After each Retry, the board shows the same refusal and its reason, not an unchanged generic notice. ### Recovery refusal classes | Retry class | Failure codes | Automatic recovery | Board Retry | | --- | --- | --- | --- | | `retry_transient` | `provider_unavailable`, `export_interrupted` and other transient codes | Retries with backoff, up to 5 retries in 30 minutes | Starts a new recovery round | | `recheck_evidence` | `agent_stop_unconfirmed`, `workspace_writer_active`, `effect_unresolved`, `copyback_stop_unconfirmed` | Checks the evidence again later | Starts a new round (the copy-back stop case has its own wait text) | | `use_alternative_strategy` | `historical_source_missing`, `source_unavailable` (no recorded source) | Stops. The retained-source fallback can continue from the last save | Starts a fresh attempt | | `service_required` (deterministic) | `allocation_ownership_ambiguous`, `source_binding_invalid`, `snapshot_verification_failed`, `archive_unsafe`, `source_unavailable` (recorded source gone) | Stops after one attempt. The same saved state gives the same result | Runs the same check again. The board now says this and names the reason | ## Verification - Red before the fix: `recovers saved work after 1|2 warm handoffs that began with another task in the same workspace` failed with `expected 'allocation_ownership_ambiguous' to be null`. - Green after the fix: `npx vitest run src/services/workspace-recovery-warm-handoff.test.ts src/services/workspace-recovery-source.test.ts` passed (60 tests). - Safety cases that stay fenced: a predecessor lease that names a different task than its run (`lease_task`), and a cross-task predecessor with no shared workspace (`unshared_workspace`). The existing `task` fault (another task with no workspace) also stays fenced. - Sibling and retry suites passed: environment runtime and orphan sandbox, cancelled workspace preservation, workspace recovery fallback, task retry, remote workspace lifecycle, explicit native continuation, native cleanup admission, and remote retry admission. - Copy, red before and green after: `system-recovery-state.test.ts` (`states a deterministic refusal plainly and never claims retries that did not happen`), and the end-to-end `names a deterministic saved-files refusal behind Retry when no fallback can continue, and another Retry shows it again` in `workspace-recovery-preparation-gates.test.ts`. On the old code the wait reason was `saved_work_recovery_failed`. With the change it is `saved_work_recovery_refused`, with the refusal reason, for both Retry clicks. - Updated expectation: the board handoff in `workspace-recovery-retry-copyback.test.ts` now expects the refusal text for its `source_binding_invalid` source (79 tests pass). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed. - I compared the real incident's lease chain with every predecessor condition through read-only diagnostics. All eight leases match on agent, sandbox, workspace, and remote directory, and each predecessor run saved a snapshot at an older epoch. ## Risks - Recovery now trusts a predecessor from another task in the same execution workspace. Acquisition already allows that handoff, and recovery still requires the predecessor's own run and finalization to be settled and saved. A workspace is the unit that owns the files, so this does not let one workspace read another workspace's files. - I could not read one predecessor's finalization row in the real incident. If that row is not in a settled phase, recovery still refuses with `allocation_ownership_ambiguous`, which is the old behavior. - `snapshot_verification_failed` is labeled `service_required` at its 19 call sites, so it now shows the refusal text. If one of those sites is really transient, the text says "Retry runs the check again", which is still true. - A second cause kept the same task stuck: a remote snapshot save that a restart interrupted stayed `running`, and that blocks the retained-source fallback. That fix comes in a separate pull request. - No schema or API change. The new wait reason is a new string in an existing field. The UI shows the message as it receives it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only browser diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:53:37 -06:00
Nicky LeachandPaperclip	052c638b51	fix(environments): honor stopped-task proof after warm preparation (#20289) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm plugin Runner retains its task for later runs. > - A follow-up can find that the retained task has stopped. > - The resume path already asks settle to check status before it sends stop. > - A saved startup marker disabled this order, even after warm preparation had finished. > - This pull request applies the same stop-proof check to all observed-stop releases. > - The next run skips a repeated stop when status proves both execution and required preparation have stopped. ## Linked Issues or Issue Description Fixes: #20224 Refs #20203 ## What Changed - Remove the startup-marker exclusion from the existing status-first path. Keep the existing shared confirmation check. It still requires `preparationStopped: true` for startup work. - Keep stop followed by status when the first status lacks the required proof. Keep the startup drain, credential writeback, complete, and cleanup failure handling. - Add retained-task coverage with the startup metadata that warm preparation saves. Cover a startup cleanup whose first status lacks preparation proof. - Update the plugin task lifecycle contract. ## Verification - Local macOS, source base `3e8744fa91`: from `server`, `pnpm exec vitest run src/__tests__/environment-task-runtime.test.ts src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts` passed: 3 files, 82 tests. The integration tests use real embedded PostgreSQL and controlled plugin results. - After restoring the fix, the two affected files passed again: 73 tests. - Independent parent verification on commit `19f720ff68`: the two files filtered to the changed startup-proof and retained-task cases passed (4 tests). Independent source review found no actionable issue. - Baseline proof: with only the source condition restored to upstream, the new retained-task test with saved startup metadata fails because it sees an extra `stop`. The case without that metadata passes. The source fix was then restored. - The retained-task regression asserts exactly `environmentResumeLease`, `status`, `complete`, and `environmentAcquireLease`, with both task operations bound to the old task. The startup test also proves that missing preparation proof sends `stop` and a second `status`. - Test setup: `node scripts/agent-preflight.mjs --skip-rust-debug --explain` installed dependencies, then stopped because Cargo is unavailable on this Mac. Initial test collection failed because the plugin SDK output was absent. `pnpm --filter @paperclipai/plugin-sdk build` built that required dependency and the shared package; the targeted test run then passed. - Standard Greptile reviewed commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a` at 5/5 with no review threads. [Review](https://github.com/paperclipai/paperclip/pull/20289#issuecomment-6743829614). - Full local checks were not run, as requested. [GitHub PR workflow 42644023868](https://github.com/paperclipai/paperclip/actions/runs/42644023868) passed on final commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a`. It supplied broad typecheck, build, and test coverage. The separate Snyk status passed. Optional Storybook checks were skipped. - The change removes one provider stop call on the reported path. The issue's measured eight calls would become seven if other calls stay the same. The earlier five-call target is not claimed. No deployed latency measurement is claimed. ## Risks - A provider that cannot prove preparation stopped still gets stop and status. That fallback can cost one extra status call, as the existing status-first contract already allows. - A status failure keeps cleanup pending and blocks replacement. No termination proof or credential handoff is skipped. - No schema, API, dependency, or timer change. No added work on successful task reuse or ordinary task dispatch. - Live provider acceptance remains unrun. The local evidence checks orchestration and stop-proof handling. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository inspection, shell tools, and code execution. The exact serving model ID and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 07:44:19 -08:00
DottaandClaude Opus 5.5	dc9ad17515	perf(storage): time workspace snapshot save and restore steps and count storage retries (#20280) ... ## Thinking Path - Cloud durable storage is a major source of failures and slowness on staging. For example, a staging task failed when saving files after automatic retries. - Before measuring or fixing it, we need to see where storage time goes. Today a saved-workspace restore before the turn appears only inside the coarse `workspace_environment_realize` phase. - After the turn, the save's encrypt, put, verify and multipart throttle backoff have no timing at all. Only `captureMs`, `transferMs`, `verifyMs`, `uploadMs` and `publishMs` exist, in `saveTiming`. - Storage retries happen in the SDK and in the multipart uploader, invisibly. - This adds the measurement first, so the storage-mode chaos waves (MinIO behind a fault proxy) can report save and restore p50/p95 and point at the dominant wait. ## Linked Issues or Issue Description **What happened?** Durable-storage saves and restores were opaque in the run log and save timing. A slow or repeatedly retried storage call could not be told apart from slow capture, encryption or sandbox work. **Steps to reproduce** 1. Run a remote task with S3-compatible durable storage, so it saves after the turn and restores on the next run. 2. Inspect the run log and the `remote_workspace_snapshot` operation's `saveTiming`. The restore's download, decrypt and verify, and the save's put, verify and throttle backoff, have no timing. **Expected behavior** Each storage step has a duration, and storage request retries and backoff are counted. ## What Changed - `server/src/storage/request-stats.ts` (new): an async-local collector of storage request statistics. Counts only: - `requests`, `attempts` and `failedRequests`; - `retryDelayMs` (SDK waits); - `parts`, `throttled` and `backoffMs` (the multipart uploader's waits). - `server/src/storage/s3-failure.ts`: `withS3FailureDiagnostics`, which wraps every S3 send, records each request's SDK `$metadata.attempts` and `totalRetryDelay`, on success and on failure. - `server/src/storage/s3-multipart.ts`: records each uploaded part and each resend wait, flagged throttled when the backend returned 503, 429 or SlowDown. - `server/src/services/remote-workspace-snapshot-storage.ts`: - Each step is a `storage.snapshot.*` run phase: - save: `prior_attempts`, `encrypt`, `digest`, `put`, `verify`; - restore: `download`, `decrypt`, `verify_plaintext`, `publish`. - A caller can pass a `timing` record that receives the step durations, the bytes and the request statistics. - `server/src/services/remote-workspaces.ts`: - The save passes a timing record, kept in `saveTiming.storage`. - The sandbox unpack during a restore is its own phase, `workspace.snapshot.sandbox_restore`. - `execution-preparation-timing.ts`: the new phase names. - `doc/run-log-events.md`: a new "Durable storage timing" section. Because the restore runs inside run preparation, its steps of 50 ms or more become `run.performance.span` rows with no extra plumbing. This is run-log and instance-database data only; there's no Telemetry change. ## Verification - `server`: `storage-s3-multipart-backoff`, `storage-s3-multipart`, `storage-s3-multipart-throttle`, `storage-s3-failure-diagnostics`, `remote-workspace-snapshot-storage` and `execution-preparation-timing` tests: 86 passed. They include: - collected stats for a throttled multipart upload: 12 requests, 7 failed, 7 throttled, 35.5 s backoff and 3 parts; - a save and restore that fill every step duration and the byte count. - `remote-workspaces.integration.test.ts`: 112 passed. Its `saveTiming` assertion now includes `storage`. - Server typecheck: no errors in the touched files. ## Risks - **Overhead:** one async-local lookup per S3 request, plus timestamps per snapshot step. - **Collector scope:** the collector only exists inside a snapshot save or restore. Other storage callers are unaffected; they record nothing. - **`saveTiming`** gains a `storage` object. Readers that compare it exactly (one test, updated) will see the new field. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:41:53 -06:00
DottaandClaude Opus 5.5	3ef45fdd92	fix(recovery): let a new task wait for its manually paused assignee (#20281) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A person can pause an agent. Tasks assigned to a paused agent wait until the person resumes it. > - Stranded-task recovery looks for assigned work that cannot continue and moves it to `blocked` for the board. > - A task created for a manually paused agent has no run yet. Recovery saw an assignee that is not invokable and moved the new task to `blocked` a few seconds after creation. > - The person paused the agent on purpose. The blocked task and its "the board must choose the next action" comment fight that decision, and the person must unblock the task by hand after resume. > - This pull request lets new `todo` work with no run wait while a manual pause lasts. The existing assigned-todo dispatch starts it when the agent is resumed. > - The benefit is that work queued for a paused agent starts on resume, as the pause promises. ## Linked Issues or Issue Description **What happened** A chat evaluation pauses an agent (Morgan) and then runs `paperclipai issue create ... --assignee-agent-id <Morgan> --status todo`. The CLI returns the task with status `todo`. Within about five seconds the task changes to `blocked`, with the system comment "Paperclip cannot safely continue automatic recovery because the original assignee is not invokable. The source assignment is unchanged and the board must choose the next action." The server log shows only the expected 409 "Agent is not invokable" for the assignment wake. **Expected behavior** The task stays `todo` while the person keeps the agent paused. When the person resumes the agent, the task starts once. A comment and a `blocked` status are not added. **Steps to reproduce** 1. Pause an agent from its page. 2. Create a `todo` task assigned to that agent. 3. Wait for the stranded-task recovery sweep. See that the task is `blocked` with the comment above. ## What Changed - `server/src/services/recovery/service.ts`: in `reconcileStrandedAssignedIssues`, skip a `todo` task with no run when its assignee is paused with a `manual` lifecycle hold. The existing rule above it already skips work whose run the pause cancelled. Other pause causes, such as a budget hold, keep the existing escalation. - `server/src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: a PostgreSQL test. The task stays `todo` with no comment and no run during two recovery sweeps. After resume it starts exactly once and finishes. A budget-held assignee still escalates. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 2 passed. Without the fix, the first test fails with `expected 'blocked' to be 'todo'`. - `npx vitest run src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-refused-assignment-wake.test.ts`: passed. - `src/__tests__/issue-tree-pause-resume-run.test.ts` had timing failures on this low-memory machine both with and without this change (different tests each time). Those tests do not pause agents, so this change does not reach them. CI runs them on this branch. ## Risks - Low risk. The skip needs all of these: the agent is paused, it has a `manual` hold, the task is `todo`, and the task has no run. Work that already ran keeps the current rules. - No persisted state changes shape. State saved by the previous release carries forward: a task that the previous release already moved to `blocked` stays `blocked`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20143 (a pause's stop never blocks the task behind an incident), #9998 (open: reject paused agents as assignment targets). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:26:20 -06:00
DottaandClaude Opus 5.5	6db0bb9776	fix(chat): tell agents which CLI commands a person runs in their own terminal (#20267) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The chat guidance in `server/src/services/agent-conversations.ts` tells every chat agent how to answer. > - People often ask a chat agent for a terminal command: create a task, download a plan, inspect or stop a run, or point the CLI at a company. > - The guidance did not name the Paperclip CLI. Agents guessed: they did the work themselves, wrote `curl`, used placeholder company ids, or invented an API address. > - The person then has to fix the command, or gets a file in the agent's workspace instead of on their own machine. > - This pull request adds one guidance paragraph. It names the supported `npx paperclipai` commands, asks for real ids from the conversation, and asks for the person's own Paperclip URL as a placeholder. > - The benefit is a command the person can run as it is, with the right company and the right actor. ## Linked Issues or Issue Description **What happened** Three chat evaluations (cli-operations) asked for terminal commands. "Show me how to point my terminal at the Acorn company." got `context set` with `<acorn-company-id>` and an invented `http://localhost:3100`, although the agent can read its company id. "Download this task's plan from the terminal as a file." got a file saved in the agent's own workspace and an attachment link, with no terminal command. "Show me the command to inspect Morgan's last run and stop it if it is still running." got `curl` with environment variables that exist only in the agent's runtime. **Expected behavior** The agent gives `npx paperclipai` commands that the person runs in their own terminal. The commands use real ids that the agent can read in the conversation. The Paperclip address is the placeholder `<your Paperclip URL>`. The agent says which commands need a board user. **Steps to reproduce** 1. Open a chat with an agent. 2. Send "Download this task's plan from the terminal as a file." 3. See that the agent saves the file itself and gives no terminal command. ## What Changed - `server/src/services/agent-conversations.ts`: add a chat guidance paragraph for terminal and CLI requests. It names the supported commands for task creation, document download, run read and cancel, and context profiles. It asks for one command per line, real ids, and the `<your Paperclip URL>` placeholder. - `cli/src/__tests__/chat-cli-guidance.test.ts`: a new test reads the guidance and resolves every `npx paperclipai` command against the registered commander commands. It fails on an unknown command, flag or missing argument. - `server/src/__tests__/agent-conversations.test.ts`: check that the paragraph is in the directive and in the chat prompt. ## Verification - `cd cli && npx vitest run src/__tests__/chat-cli-guidance.test.ts`: 3 passed. - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts`: 80 passed. - The chat evaluations `cli-operations.choose-cli-context`, `export-document-cli` and `inspect-run-cli` run with this commit on an evaluation branch. Results are added in a comment. ## Risks - Low risk. The change adds prompt text and tests. Routes, permissions and responses do not change. - The chat prompt is about 300 words longer. - If a CLI command or flag changes, the new CLI test fails until the guidance changes too. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20029 (terminal sign-in guidance in the API catalog). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:26:13 -06:00
Pavel-GuseynovandClaude Opus 5.5	4d297b998a	fix(server): give remote MCP tool calls a 30 second default deadline and record stream failure evidence (#19768) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The tool gateway calls remote MCP servers over Streamable HTTP, and a server can answer `tools/call` with a `text/event-stream` body > - Such a server can send progress notifications before the response, and a slow tool can take more than 10 seconds > - The gateway aborts a remote MCP `tools/call` that uses the default deadline after 10 seconds, the general tool default > - When the call fails, the call evidence does not show how far the stream got, or whether the cause was the deadline or a broken transport > - This pull request gives remote MCP calls a default deadline of 30 seconds, the default response timeout of the remote HTTP client, and records the failure cause and the stream progress > - The benefit is that slow remote tools complete, and an operator can tell a deadline from a peer reset in the call evidence ## Linked Issues or Issue Description Refs #10147. That issue reports that a named MCP gateway aborts remote tools after 10 seconds and ignores the connection timeoutMs. This pull request raises the default deadline of a remote MCP `tools/call` to 30 seconds. It does not make the named gateway read the connection timeoutMs. Related: #11910 (open) keeps a connection callable after one timed-out tools/call. On current master a timeout and a JSON-RPC response error already keep the connection healthy, and this pull request keeps that policy. I searched pull requests and issues for "event stream tools/call", "SSE MCP tool call hang", "remote MCP timeout stream", "mcp text/event-stream" and "tool_timeout remote MCP". The only related open pull request is #11910. ## What Changed - `server/src/services/remote-http-fetch.ts`: the default response timeout is exported as `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` (30,000 ms, unchanged value). - `server/src/services/tool-gateway.ts`: - A remote MCP `tools/call` that uses the default deadline now gets `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` instead of the general tool default of 10 seconds. The Railway `run-command` budget and an explicit `timeoutMs` keep their values. - The call evidence records the response status, content type and request id before the body is read. A body that fails partway records the bytes received so far as its size. - A failed call records `failureKind` (`invocation_timeout`, `protocol_failure` or `transport_failure`), the deadline and whether it was aborted, the stream progress, and the error with its cause chain. Each string in the chain is redacted with `redactSensitiveText`, and a cyclic cause chain is cut with a "[Circular cause]" marker. - A deadline abort is a 504 `tool_timeout` and starts a fresh MCP session on the next call, as before. The connection health policy of master does not change. - A deliberate stop of a call (`elicitation_required`, `provider_interaction_required`, `oauth_insufficient_scope`, `oauth_refreshed_retry_required`) keeps its own reason when the deadline passes while it is recorded, as on master. An error that is already a gateway error also gets a failure kind: `tool_timeout` is `invocation_timeout`; a malformed, invalid or oversized response is `protocol_failure`; and `remote_http_*` (for example a DNS failure) and `mcp_remote_fetch_failed` are `transport_failure`. - A non-success response body is read as a stream, so a body that fails partway records the bytes that arrived. The reader stops at the size limit and cancels the body, as the `tools/call` reader does. #19785 makes the same reader change for the context path. - `server/src/services/mcp-http.ts`: `readMcpHttpResponse` takes an `onProgress` callback. It reports bytes, chunks and events received, whether the matching response arrived, and whether the stream ended. As on master, the read returns as soon as the response with the matching id arrives. - Tests: - `server/src/__tests__/mcp-http.test.ts`: a partial event followed by a peer error keeps the progress that was consumed, and releases the reader. - `server/src/__tests__/tool-gateway.test.ts`: six scenarios run against a remote server that streams one progress event first: slow success after 12 seconds, the default deadline, an explicit 1,500 ms deadline, a peer reset, a cyclic error cause, and a primitive error value. - `server/src/__tests__/tool-gateway.test.ts`: three regression tests: a tool question sent after the deadline passed keeps `elicitation_required`; a DNS failure records `transport_failure`; and a 503 body that fails partway records the bytes that arrived. All three fail on 717e6c77c and pass on this head. - `server/src/__tests__/tool-gateway-service.test.ts`: when the call deadline passes during a forced Vercel token refresh, the call keeps the 409 `oauth_refreshed_retry_required` stop. `tool-gateway.ts` adds this reason to the deliberate stops, so the deadline does not change it to a 504 `tool_timeout`. ## Verification Head SHA: `25c6e703d1ded895989e9d2fcd6d6352a4f851f5`. It is the previous head `7836447f2` plus commit `25c6e703d`, which keeps the `oauth_refreshed_retry_required` reason when the call deadline passes during a token refresh. At this head, run from `server` with pnpm 9.15.4: `pnpm typecheck`: no errors; `mcp-http.test.ts`, `tool-gateway.test.ts` and `tool-gateway-service.test.ts`: 307 passed. Without the one-line change in `tool-gateway.ts`, the new test "keeps the explicit retry reason when the call deadline passes during the Vercel token refresh" fails: it receives 504 `tool_timeout`, not 409 `oauth_refreshed_retry_required`. Previous head: `7836447f24e2e6e9e0c402d4644d1f7f69b75cb8`. It is an ordinary merge of master `76994c50275dbc005cc248e11189a38baa21719f`, which includes #19877, into the previous head `00150e59c`. At this head, run from `server` with pnpm 9.15.4: `pnpm typecheck` and the test commands below. Results: typecheck: no errors; the 2 test files above: 116 passed. Previous head: 00150e59c1. It merges master 18b6b71d8. Run from `server`: ```sh pnpm exec vitest run src/__tests__/mcp-http.test.ts src/__tests__/tool-gateway.test.ts ``` - With this branch: 116 passed. With tool-gateway-service.test.ts and remote-mcp-socks-proxy.test.ts added: 315 passed in 4 files. - With `services/mcp-http.ts`, `services/remote-http-fetch.ts` and `services/tool-gateway.ts` from master and the test files from this branch, measured at 717e6c77c: 7 failed, 106 passed. The failing tests are the six "records remote SSE ... without confusing a call deadline with a broken transport" scenarios and "retains consumed SSE progress when a partial event fails and releases the reader". - The existing master tests "keeps a remote MCP connection healthy and starts a fresh MCP session after a tool call timeout" and "... after a response body timeout" pass with this branch. - Typecheck: `pnpm --filter @paperclipai/server typecheck` reports 29 errors, all "Cannot find module @paperclipai/plugin-sdk" and the implicit any that follows, in the sandbox provider plugin packages of my local install. None is in a file that this pull request touches. - I ran the tests with an embedded PostgreSQL from the repository dependencies. The full upstream suite runs in CI. I ran only the test files named above. ## Risks - Medium risk for slow tools: a remote MCP call that uses the default deadline can now run for up to 30 seconds instead of 10 before it fails with `tool_timeout`. An explicit `timeoutMs` still applies. - The call evidence is larger: it can hold a redacted error chain and its stack traces. All strings go through `redactSensitiveText`. - No automatic replay: a timed-out `tools/call` is not sent again, because it can have changed data in the remote app. ## Model Used - Original change: the model that wrote it was not recorded. Port to current master, the adaptation to the connection health policy of master, and this description: Claude Opus 5.5 (claude-opus-5-5), with tool use and terminal code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing documentation change needed) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:19:08 -06:00
DottaandClaude Opus 5.5	ac0d0374f2	perf(issue-detail): measure task opens on real instances (#20276) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description **Problem or motivation** Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. **Proposed solution** Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. **Alternatives considered** Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. **Roadmap alignment** This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:06:37 -06:00
DottaandClaude Opus 5.5	fb88a89f04	fix(chat): tell chat agents how to link a case page while Cases is enabled (#20265) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers questions about company records and links each record it cites > - Chat guidance gives link formats for tasks, approvals, runs and agent profiles > - Cases (an experiment) have their own page at `/<prefix>/cases/<identifier>`, but the guidance does not name it > - So an agent that answers a question about a case's fact history names the case but cannot link it > - This pull request adds a Cases directive to chat guidance, only while the Cases experiment is on > - The benefit is that a person can open the case, where its facts and audit history are, from the reply ## Linked Issues or Issue Description **What happened** In a chat eval with Cases enabled, a person asked who changed a case's diagnosis. The agent read the case audit and the task's document revisions and answered correctly. It wrote "case COM-C1" as plain text. It linked only the task. The guidance in the chat prompt has no link format for a case, and the agent does not guess routes. **Expected behavior** When a reply concerns a case (its facts, fact-change history or handoffs), the agent links the case page as a Markdown link, for example `[RUN-C1](/<prefix>/cases/RUN-C1)`, and also links the tasks it cites. When the Cases experiment is off, the prompt does not mention cases. **Steps to reproduce** 1. Enable the Cases experiment. 2. Create a case with a primary task, and change one of its facts. 3. In Agent Chat, ask "Who changed the diagnosis, and what did it say before?" 4. Before this change, the reply names the case identifier but has no link to the case page. ## What Changed - `server/src/services/agent-conversations.ts`: new `CASES_CHAT_DIRECTIVE`. It says what a case is, names the `/<prefix>/cases/<case identifier>` page, and asks for a Markdown link when a reply concerns a case. - `server/src/services/heartbeat/task-markdown.ts`: new `casesEnabled` input. A conversation prompt includes the directive under "Cases:" only when it is true. - `server/src/services/heartbeat/task-context.ts`: sets `casesEnabled` from the `enableCases` experimental setting, the same way as `statusCardsEnabled`. - `server/src/services/heartbeat/task-markdown.test.ts`: new test. The directive appears for a standing chat with Cases on. It does not appear with Cases off or for a non-chat task. The test also checks that the board route `cases/:caseIdentifier` exists in `ui/src/App.tsx`. ## Verification - `cd server && npx vitest run src/services/heartbeat/task-markdown.test.ts` passes (18 tests). - `npx tsc --noEmit -p server` passes. - The `cases.case-history` chat eval uses this guidance. Its rerun is on branch `chat-eval/case-unclear-9`. ## Risks - Low risk. The change adds prompt text only when the Cases experiment is on. It does not change any API or permission. - The prompt for chats on instances with Cases enabled is slightly longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M context, extended thinking, with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:03:16 -06:00
DottaandClaude Opus 5.5	1004e55180	fix(recovery): stop stall wakes from failing setup on a held workspace (#20273) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, check for proof that its run would be refused. The proof is that the assignee's own latest run on the task failed setup with the unfinished-writer error, and the bound workspace hold still refuses a run. Then do not wake it; count it as deferred and re-check at most hourly. A different assignee (for example a controller-only agent that never uses the workspace), a later run, or a settled hold lets the wake through. Reservation already tried and failed the unchanged-writer release for that run, so the sweep does not repeat that proof. - `blocker-stall-wakeups.ts`: count failed runs that carry the stall (the keyed wake's run and keyless resource-wait retries that keep its context). Retry after 10, then 20 minutes, and stop after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `remote-workspace-writer-hold.ts` / `remote-workspaces.ts`: share the unfinished-writer error text as one constant. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 22/22. Added after review: a waiter with no refused run, or a reassigned waiter, is still woken; failed resource-wait retries count toward the limit. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. The parent's own earlier run failed setup on the hold, as on staging. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:28:21 -06:00
DottaandClaude Opus 5.5	2dd7a98753	fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, read its bound remote workspace hold. When the hold refuses a run, do not wake it; count it as deferred and re-check at most hourly. A workspace with an unchanged-release baseline is left to reservation, which can release it. - `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried after 10, then 20 minutes, and not after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:28:04 -06:00
DottaandClaude Opus 5.5	1b09920a64	fix(issues): refuse agent status changes that detach another run's live execution (#20271) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that an agent run executes holds a checkout lock and an execution lock (`checkoutRunId`, `executionRunId`) > - A status change away from `in_progress` clears these locks. It does not stop the run that holds them > - Agents cannot stop another agent's run: run cancel and the task-tree pause are board-only > - But an agent that can edit a task (for example a lead with the manage-active-checkouts grant over its report) can change its status. The run then keeps running and is no longer bound to its task > - In chat evals, a lead "paused" a launch tree. It moved running subtasks to backlog. The runs kept running, detached > - This pull request refuses an agent's status change when a different queued or running run holds the task's lock. The denial is final, links the task, and names the person's controls > - The benefit is that agents cannot interrupt another agent's run through a status edit, and runs are not left orphaned ## Linked Issues or Issue Description Refs #20261 (chat guidance sends a task-tree pause to the person's Pause subtree control). Refs #19394 (resume restarts the work a subtree pause interrupted). **What happened** An agent changed the status of a task while another agent's run executed it. The request succeeded. The server cleared `checkoutRunId` and `executionRunId`. The run continued, but it was no longer bound to the task. In the chat evals `steering.stop-versus-pause` and `task-coordination.pause-tree`, a lead moved running subtasks to backlog to "pause" a launch. Their runs kept running, detached. **Expected behavior** An agent cannot get the effect of interrupting another agent's run. When a different live run holds the task's lock, the agent's status change is refused before any write. The refusal says it is final. It sends the person to the task and names the controls that really stop or hold the work: the run's Stop or Interrupt, or More task actions, then Pause subtree on the parent task. The lock-holding run, board users, and paths that stop the run themselves continue to work. **Steps to reproduce** 1. Create agent A and agent B, with B reporting to A (A gets `tasks:manage_active_checkouts` through the manager chain). 2. Give B an `in_progress` task. Set `checkoutRunId` and `executionRunId` to a `running` heartbeat run of B. 3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with `{ "status": "backlog" }`. 4. Before this change: `200`, the status is `backlog`, both lock fields are `null`, and B's run is still `running`. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it refuses a status change when `executionRunId` or `checkoutRunId` names a different run in the same company that is `queued` or `running`. It runs in `PATCH /issues/:id` after the final status and assignee are known (execution-policy transitions included) and before any run stop or write. It also runs in recovery resolution when the source status changes. - The guard applies to any status other than `in_progress`, also to the current status, because the service clears the locks for those too. - The write is conditional on the lock the guard checked. The route re-reads the lock and passes it to `svc.update` as `expectedExecutionLock`. The service compares it under the row lock and returns `409 issue_execution_lock_changed` if a checkout landed in between. - These cases stay allowed: the lock-holding run itself, board users, requests with no status, a change to `in_progress`, `cancelled` (the route cancels the run), reassignment (the route stops the run first), `done` through PATCH while a Runner goal is active (terminalization pauses the goal and stops the run; recovery resolution does not get this exception), and a lock holder that already finished. - `packages/shared/src/issue-write-denial.ts`: new denial code `issue_status_change_live_run` (409). The copy says the refusal is final, links the task as `[identifier](/prefix/issues/identifier)`, and names Stop or Interrupt and More task actions, then Pause subtree. A new optional `issuePath` context field carries the link. The UI notice renders it through the existing generic path. - `doc/SPEC-implementation.md`: documents the rule in the checkout contract section. - Paths checked with no change needed: there is no bulk issue update route; tree holds and run cancel are board-only; `POST /issues/:id/release` already lets only the assignee's checkout run release; API-catalog actions call the same PATCH route; MCP status changes already refuse with `MCP_ACTIVE_EXECUTION`. ## Verification - Red on master, green with this change: - `server/src/__tests__/issue-stale-execution-lock-routes.test.ts` (embedded Postgres, real issue service): a manager moves its report's running task to backlog. On master: `200`, status `backlog`, `checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409 issue_status_change_live_run`, row unchanged, run still `running`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued lock holder, and a repeated current status (red on master). Allowed cases: field edits, the lock-holding run, board, `cancelled` (run is cancelled), and finished lock holders; an allowed change carries `expectedExecutionLock`. - Same embedded-Postgres file: `svc.update` with an `expectedExecutionLock` that differs from the row returns `409 issue_execution_lock_changed` and leaves the row unchanged. - Also passed locally: `issue-closed-workspace-routes`, `issue-comment-reopen-routes`, `issue-recovery-actions`, `issue-scheduled-retry-routes`, `low-trust-red-team-routes`, `issue-update-comment-wakeup-routes`, `external-object-routes`, `packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`. - `tsc --noEmit` shows no errors in the changed files. Full typecheck, build and test are left to CI. ## Risks - Behavior change for agents: a lead or CEO that used status edits to take over a report's running task now gets a 409. The supported paths (comment, cancel, reassign, or the person's Stop or Pause subtree) still work. - A queued run that holds the lock also blocks the change. This is deliberate: the queued run would otherwise start later with no task bound to it. - Agents can still cancel another agent's run by cancelling its task. That is existing, documented behavior ("task cancellation cancels the intent"), and this PR does not change it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:21:40 -06:00
DottaandClaude Opus 5.5	6459488984	fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, a remote (Daytona) run can resume by reattaching to its surviving sandbox runner. If provider recovery fails and the checkpoint allows it, the run opens a replacement provider session instead. > - Before a replacement, the executor archives the local controller state and the remote session root as a continuity break, then opens a new session. > - On a reattach backend, the replacement still adopted the archived runner and asked for its old authority, so it failed. The archive had already moved the remote state, so the next retry could not verify the runner and the issue waited 17 minutes on an unconfirmed stop. > - Review showed that a reattach backend fixes many launch inputs once at creation (runner binary, provider pack and companions, artifacts, live-recovery mode, authority, spawn barrier), so resetting them after the archive does not scale. > - This pull request builds the replacement on a new backend for the same run, created under a `bootstrap_incomplete` claim after the archive, so every launch input resolves through the ordinary fresh path. > - The benefit is that a failed reattach after a restart falls back to a working replacement instead of a stuck issue. ## Linked Issues or Issue Description **What happened?** In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1 (no rest within budget). A K2 drain suspended run a53dd948's sandbox runner (pid 589) during startup. On restart the run claimed `reattach_remote_runner`. The adopted runner authenticated and restored, then its connection closed, and provider recovery failed. The runtime opened a replacement: `provider.session.archive_before_replacement` archived the local and remote state, then the replacement failed with "PRP provider resume state is unavailable". The same-run retry at 10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the remote `runner-state.json` had moved), and file preservation deferred on `agent_stop_unconfirmed` until the wave ended. **Expected behavior** After the continuity archive, the replacement opens a new runner for this run, as it does for every other claim kind. **Steps to reproduce** 1. Run a remote (Daytona) native run and restart the server so the run claims `reattach_remote_runner`. 2. Make provider recovery fail on the adopted runner with a checkpoint that allows replacement. 3. See the replacement open on the adopted runner's authority and fail after the archive. **Paperclip version or commit** master at 465da4227a ## What Changed - `native-session-executor.ts` `openReplacementSession`, on a `reattach_remote_runner` backend: archives the continuity state (local and remote, as before), then calls `createRunnerdBackendWithinSessionClaim` for the same run with the claim re-kinded to `bootstrap_incomplete` (minus `remote` and `stagedAttachment`) and without warm recovery, and opens the replacement on that backend. Root migration then quarantines the empty archived root (the #20241 rule) and every launch input resolves fresh. - A backend that adopts only through warm recovery (no claim to re-kind) throws `native_runner_replacement_requires_fresh_launch` **before** archiving, so its saved state stays intact. - Tests: a classic Daytona reattach backend for `codex` and `acpx` (Claude). The replacement comes from a second backend: the original backend's `openSession` is never called, and the new transport has no adopted runner, uses `turn-<runId>`/`item-<runId>`, has a new still-pending spawn barrier, and (for `acpx`) carries the packaged provider paths. The bundled-image replacement test now expects a new backend and its own pack admission, not the recovered session's cold admission. ## Verification - Red before the fix: both new replacement tests fail on master. They pass with the fix. - `native-session-executor.test.ts`, `native-recovered-owner-followup.test.ts`, `native-harness-filesystem-independence-continuation.integration.test.ts`: 1,398 passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Medium. The replacement now constructs a second backend inside the same session claim. The new backend's tool-authority epoch supersedes the reattach backend's, which is being abandoned anyway. - A warm-recovery-only replacement now fails with a named error instead of archiving and then failing. Its saved state stays intact for the next recovery. - Previous-release story: no saved state or schema changes. - Not covered: why provider recovery on the suspended, adopted runner failed in t36 in the first place. This PR makes the fallback work. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 08:12:24 -06:00
DottaandPaperclip	a3a25ce6c9	fix(workspaces): stop running Paperclip's worktree setup before every agent turn (#20269) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent turn starts, the harness prepares the task's git worktree > - The harness runs a repository's `scripts/provision-worktree.sh` when the project sets no `provisionCommand` and the file exists > - The harness runs it on every turn, both when it creates the worktree and when it reuses it > - In the Paperclip repository, that script writes the worktree's isolated instance config and runs `pnpm install` > - Its "install needed" check compares against every `node_modules` directory in the base checkout, including other worktrees, so it reinstalls on every turn > - This pull request renames the Paperclip script so that the harness default does not find it in this repository > - The benefit is that Paperclip agent turns no longer wait on a 6–9 second install. Other repositories keep the harness feature. ## Linked Issues or Issue Description **What existing behavior does this improve?** Time to first token for agent turns in managed worktrees of the Paperclip repository. **Subsystem affected** Execution workspace provisioning (`server/src/services/workspace-runtime.ts`) and the repository script `scripts/provision-worktree.sh`. **Current behavior** Every agent turn in a Paperclip worktree first runs `bash ./scripts/provision-worktree.sh`. In a local test, a `pnpm install --frozen-lockfile` with nothing to change took 6.1 s and 6.3 s, and a full install from a warm store took 9.0 s. The script's fingerprint skip does not prevent this, because the script also requires every base-checkout `node_modules` path (up to depth 4) to exist in the worktree. On a developer machine that list had 1,735 entries, including other worktrees under `.claude/worktrees/` and `.worktrees/` and pnpm's internal `node_modules/.pnpm/*/node_modules`. A managed worktree never contains the other worktrees' paths, so the script always decides to reinstall. **Proposed behavior** The Paperclip repository does not ship a file at `scripts/provision-worktree.sh`. The setup script becomes `scripts/setup-paperclip-worktree.sh`. People can run it by hand or set it as the project `provisionCommand` to opt in. The harness default stays for other repositories. **Reason and benefit** Agents install dependencies when they need them (`node scripts/agent-preflight.mjs`, as `AGENTS.md` says). Many turns never run code. Instance setup stays available on demand: `scripts/provision-worktree-runtime.sh` runs the setup script when a dev server starts in a worktree that has no config. Database seeding was already deferred. **Breaking changes** A project that explicitly sets `provisionCommand` to `bash ./scripts/provision-worktree.sh` for this repository will fail with "file not found". Change it to `bash ./scripts/setup-paperclip-worktree.sh` or clear it. ## What Changed - Rename `scripts/provision-worktree.sh` to `scripts/setup-paperclip-worktree.sh`. The content is unchanged. - `scripts/provision-worktree-runtime.sh` calls the new name when the worktree has no config. - Tests read the real script from its new path (`provision-worktree-self-heal`, `workspace-runtime`, `worktree-seed-server-spawn`). Server fixtures still copy it into their test repositories as `scripts/provision-worktree.sh`, so the harness default stays covered. - `doc/DEVELOPING.md` says that the Paperclip repository deliberately has no `provision-worktree.sh`, and tells people how to opt in. ## Verification - `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs`: 33 passed, 1 skipped (the skip already existed). - `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts src/__tests__/heartbeat-workspace-source-trust.test.ts src/__tests__/execution-workspace-reopen.test.ts`: 242 passed on the rebased head. - `cd server && npx vitest run src/__tests__/worktree-seed-server-spawn.test.ts`: 2 passed. - Manual: in a Paperclip worktree, start an agent turn. The run log does not show a `workspace_provision` operation for the script. ## Risks - Explicit `provisionCommand` settings that name the old path will fail. See "Breaking changes". - Open pull requests that edit `scripts/provision-worktree.sh` (for example #16213, #11335, #11733, #19764) will conflict on the rename. Git rename detection should carry most edits over. - An agent that runs tests in a new worktree without first running preflight will see a missing-module error. `AGENTS.md` already tells agents to run `node scripts/agent-preflight.mjs`. - The script's install check still has the bug described above. It now runs only on demand. A separate fix can address it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-09 14:03:24 +00:00
DottaandClaude Opus 5.5	33a23e0246	fix(chat): send a task-tree pause to the person's Pause subtree control (#20261) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can pause a whole task tree from the parent task page: More task actions, then Pause subtree > - The pause holds every task under the parent, stops their active runs and keeps assignees; Resume subtree restarts the interrupted work (#19394) > - The tree-control routes are board-only, so a chat agent cannot apply this pause > - Asked to pause a launch tree, chat agents imitated the pause: they moved running subtasks to backlog, cleared their execution and told the person the work was paused > - This pull request tells chat agents, in guidance they always load, to send the person to Pause subtree and not to imitate it > - The benefit is that the person gets a real hold that keeps assignments and resumes cleanly, and the reply does not claim a pause that does not exist ## Linked Issues or Issue Description Refs #20252 (marks the tree-control routes board-only in the API catalog and names the page). Refs #19394 (resume restarts interrupted work). **What happened** A person asked a chat lead "Pause the whole Launch task tree" or "Pause all work on the launch until tomorrow". In three of four evaluated runs the lead did not use or point to the tree pause. It changed the two running subtasks from in_progress to backlog, which cleared their execution, and replied "All launch work is paused". Its two native runs kept running, and the run cancel calls were denied. In one run it also said "I scheduled a check for 2026-11-10 12:10 UTC to resume the launch". No tree hold existed, so Resume subtree later had no interrupted work to restart. The agent's reasoning shows it decided early that "pause" meant backlog ("No issue-level pause endpoint. So pause = set to backlog"). It did not reach the board-only catalog entry for the tree hold, even with #20252 applied. **Expected behavior** The agent finds the parent task and replies with a Markdown link to it. It tells the person to use More task actions, then Pause subtree on that page, and Resume subtree later. It does not change task statuses or assignees, cancel runs, pause agents, or create tasks or routines to imitate the pause. It does not claim a scheduled resume, because Paperclip has none. **Steps to reproduce** 1. Create a parent task with two child tasks whose assigned agents are running. 2. In a chat with the company lead, send "Pause the whole Launch task tree." 3. Read the reply and the task activity: before this change the lead often moves the running children to backlog and says the tree is paused. ## What Changed - `server/src/services/agent-conversations.ts`: add one paragraph to the chat directive. It says that pausing, stopping or resuming a task tree is a person's control on the parent task page (More task actions, then Pause subtree, and later Resume subtree). It says what the pause does, that there is no scheduled resume, and that the agent must not imitate the pause with status, assignee, run, agent or routine changes. The agent links the parent task and replies. - `server/src/__tests__/agent-conversations.test.ts`: add a regression test for the directive text and for its presence in the built chat prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts src/services/heartbeat/task-markdown.test.ts src/__tests__/heartbeat-chat-task-link.test.ts src/__tests__/heartbeat-context-summary.test.ts src/services/native-runtime/current-wake-comments.test.ts src/__tests__/codex-local-execute.test.ts src/__tests__/claude-local-execute.test.ts` passes. - End to end: the chat evals `task-coordination.pause-tree` and `steering.stop-versus-pause` run this change on an eval branch. A human then clicks Pause subtree, and the eval checks the hold, the two acknowledged native stops, and that Resume subtree restarts only the interrupted branches. ## Risks - Low risk. The change is prompt text only. It adds about 900 characters to every chat prompt. - An agent that has a real need to change one child task's status in the same conversation can still do it when the person asks for that change. The paragraph covers requests to pause or stop a tree. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:33:40 -06:00
DottaandClaude Opus 5.5	ad440d4e91	fix(chat): name the managed-profile CLI and the dev-only labs in agent guidance (#20236) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers a person's requests with a chat agent that reads the API catalog and the chat guidance > - Two facts are missing from what that agent reads > - Claude Managed Agent profiles have no web page. A board user creates them with the `paperclipai managed-agent setup` CLI command. The catalog does not say this, so agents tell people to use "the Paperclip web app" > - Developer labs, such as `/<prefix>/dev/task-chat-lab`, exist only in a development build. Nothing says this, so agents probe the production bundle in a headless browser for many steps and then ask the person for a "dev-mode URL" > - This pull request adds the two facts where the agent already looks > - The benefit is a correct next step for the person, in fewer agent steps ## Linked Issues or Issue Description **What happened** An agent was asked to "Set up an Anthropic managed agent for this company". It correctly said the action is board-only. Then it told the board user to create the profile "in the Paperclip web app". That page does not exist. An agent was asked to "Open the task-chat interaction lab from this link". The link went to a production instance. The agent used 56 steps, mostly bash and headless Chrome, and did not tell the person that the lab runs only under `pnpm dev` in their own checkout. **Expected behavior** For managed agents, the agent names the CLI command that a board user runs. It also says that no web page exists. For developer labs, the agent says the lab is development-only. It tells the person to run `pnpm dev` in their own checkout and open the route there. It does not probe the instance. **Steps to reproduce** 1. In Agent Chat on a production build, ask "Set up an Anthropic managed agent for this company". 2. Read the reply. It points to a web page that does not exist. 3. Attach the link `http://<host>/<prefix>/dev/task-chat-lab` and ask "Open the task-chat interaction lab from this link". 4. Look at the run. The agent probes the bundle and does not give the `pnpm dev` path. ## What Changed - `server/src/routes/openapi.ts`: the GET and POST managed-agent-profile routes now say that no web page exists. The POST description now names `paperclipai managed-agent setup --company-id <companyId> --profile-key <key> --display-name <name> --api-key-secret-id <secretId> --acknowledge-retention`, the `ANTHROPIC_API_KEY` requirement, the `--probe` option and the Secrets page. - `server/src/services/agent-conversations.ts`: the chat directive has a new short paragraph. It says some developer labs, including the task-chat lab, are registered only in a development build, it tells agents not to probe, and it gives the `pnpm dev` path with the route. It also says the design guide and ux-lab pages exist in every build, so agents open those normally. - Tests for both texts in `openapi-routes.test.ts` and `agent-conversations.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/openapi-routes.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts src/__tests__/codex-local-execute.test.ts`: 142 tests pass. - After merging master (keeping master's ready-made teams paragraph) and the review fixes: `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts src/__tests__/openapi-routes.test.ts src/services/native-runtime/runner-api.test.ts`: 2164 tests pass. - The change is also cherry-picked onto the eval branch `chat-eval/case-keyword-graders`. The live Runner E2E cells for `agents.managed-profile` and `developer-labs.gated-lab` run there. ## Risks - Low risk. The change is text in the API catalog and in the chat guidance. It does not change behavior or permissions. - The chat directive is slightly longer, so each chat prompt uses a few more tokens. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:15:16 -06:00
DottaandClaude Opus 5.5	2b461b2f58	fix(native): require publication for files named under a Deliverable label (#20231) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agents finish a task with a completion report, and a server gate checks that requested files are published to the task > - The gate finds a file request only from a creation verb, such as "Write report.pdf" > - Delegated briefs often list the output under a "Deliverable:" label instead, and the gate did not know `.html` or `.css` names > - So a worker can finish a website task with `index.html` only in its own workspace, and the person cannot open it > - This pull request makes a file named under a Deliverable label a requested file output > - The benefit is that the worker gets the existing "use register_deliverable" feedback before the task closes ## Linked Issues or Issue Description No public issue exists. Bug description: **What happened** A person asked the CEO: "Make me a simple website introducing our business." The CEO delegated a task. Its brief said "Deliverable: - A self-contained static website in this task's workspace: index.html plus a stylesheet (for example styles.css)". The worker wrote `index.html` and `styles.css` in its workspace. It did not call `register_deliverable`. It cited `index.html` and `styles.css` as completion evidence, and the gate accepted the report. The task was marked done with no attachment. The CEO then told the person that the files were "in the task workspace". **Expected behavior** A brief that names a file as its deliverable is a file request. The completion gate must reject a workspace-only file and tell the worker to publish it with `register_deliverable`. **Steps to reproduce** 1. Call `explicitlyRequestsFileOutput` with the brief above. 2. See that it returns `false`, so `validateNativeDeliverableEvidence` accepts the bare `index.html` evidence ref. ## What Changed - `explicitlyRequestsFileOutput` now also returns true when a `Deliverable:` / `Deliverables` label line (plain, bold or heading), or a bullet or numbered item in the list below it (blank lines between items allowed), names a file. Relative paths such as `dist/index.html` count. - Under the label, web page files (`.html`, `.htm`, `.css`) count with the existing file types. - Items that name an input ("a summary of input.csv", "a comparison of input.csv and baseline.csv"), describe a repository change, pull request, commit or branch, forbid the named file (`do not create report.pdf`), name a file to edit in place (`update styles.css`) without asking to publish it, send the file inline (`index.html and styles.css inline in the reply`), or forbid creating files, do not count. An inline or no-upload instruction about another output in the same item (`index.html and a short summary in the reply`, `index.html; do not upload it externally`) does not cancel the named file. The repository-change check ignores file names, so `branch-report.pdf` still counts. - The sentence-based detector is unchanged. - `doc/AGENT-ARTIFACTS.md` describes the label rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-deliverable-feedback.test.ts src/services/native-runtime/native-runner-file-handoff.test.ts` (197 pass), then after the review fixes `native-deliverable-feedback.test.ts`, `native-completion-format-feedback.test.ts` and `chat-attachment-reuse.test.ts` (211 pass). The 13 first-round review cases fail on the original head. - New cases: the real delegated brief, inline labels, heading lists, numbered lists and bold labels require publication. Unlabelled text, inline-chat deliverables, input references, repository changes, text after the list ends, and "Deliverable quality ..." prose do not. - The chat eval `onboarding.first-work` exercises this path end to end on its candidate branch. ## Risks - Low risk. The change only adds a true result for labelled file outputs; an inline-only or no-file label item stays false, as the sentence detector already decided. - A labelled brief that names a file which is not a repository change now needs `register_deliverable` or an accessible work product. That is the documented delivery rule. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:15:11 -06:00
DottaandClaude Opus 5.5	68253e3175	fix(recovery): continue a run refused by an inert cleanup quarantine with no click (#20260) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native session's cleanup fails unrecoverably, the runtime quarantines it in process. The next run for that task is refused with `native_session_cleanup_quarantined`. > - A board Retry or a new message already recovers from this. `verifyStoppedNativeSessionForContinuation` proves that the old runner and providers stopped, releases the quarantine and starts a fresh session. > - Without a click nothing recovers. The automatic safe-replacement sweep proves stopped sessions only with the Codex replacement verifier, which needs the failed run's own active turn. A run refused before launch never has one, so the sweep skips it on every pass. > - Restarts must recover on their own. This pull request lets the sweep use the same continuation proof for a cleanup-quarantined run. > - The benefit is that an inert quarantine turns into a fresh-session continuation with no click, and an unproven one still keeps its board Retry. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave seed 9802 (local, G1, Claude), task t02's run 71f03e16 succeeded, but its runner cleanup could not prove it drained and was quarantined. Follow-up run 0b4b404f was refused before launch with `native_session_cleanup_quarantined`, and the task showed "Automatic recovery stopped" until someone pressed Retry. **Expected behavior** When the quarantined session's owner run and every provider process are proven stopped, recovery continues the task by itself in a fresh session, without charging an attempt. When that cannot be proven, the board Retry stays available. **Steps to reproduce** 1. Have a native run succeed while its runner cleanup is quarantined. 2. Let the next run for the task be refused with `native_session_cleanup_quarantined` before it launches. 3. Stop the old runner and provider processes, then run the safe-replacement sweep. No successor is created. **Paperclip version or commit** master at 8eb11fa8aa ## What Changed - `native-session-executor.ts`: new `verifyStoppedNativeSessionForAutomaticReplacement`. It returns the existing replacement proof, or, for a `native_session_cleanup_quarantined` run with no replacement proof, the continuation proof that the board Retry already uses (Codex and ACPX, including an inherited owner run). - `server/src/index.ts`: the safe-replacement sweep uses that verifier. - The existing sweep then retires the quarantine, records `native.stopped_text_turn_verified` and queues a successor with `forceFreshSession: true` at the failed run's attempt number, so no attempt is charged. - `quarantined-task-retry.test.ts`: "recovers a quarantined $provider source that never launched with no click…" for Codex and ACPX (Claude). A stopped owner makes the sweep queue one fresh-session successor at the same attempt. A running owner (Codex) or a live ACPX agent process queues nothing and leaves the board Retry in place. ## Verification - Red before the fix: with the replacement-only proof, the stopped case queues no successor. It passes with the fallback, and the running-owner case passes on both. - `quarantined-task-retry` (35), `native-safe-replacement`, `native-replacement-evidence`: pass. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Low to medium. The fallback applies only to `native_session_cleanup_quarantined` runs. It uses the stricter continuation proof that already gates the board Retry: owner run settled, local leases released, runner and every recorded provider pid absent, and an unchanged retained state. The sweep keeps every other gate (task ownership, failure-block receipt, attempt limit, effect inventory). - Previous-release story: no schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:12:07 -06:00
DottaandClaude Opus 5.5	1a007c4ef1	fix(accounting): settle a native run that only inherited its predecessor's runner identity as no provider work (#20250) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each finished run settles its accounting. A native run whose provider never started settles at once with no charge. Other runs wait for a usage receipt, bounded by a closure timeout. > - A company budget policy can block unpriced usage. Unpriced cost rows then trip a hard stop that pauses the company. > - A follow-up run that is refused before launch carries its predecessor's runner pid and start time. The "never started" check required a null pid, so this run waited out the closure timeout, held the board's Retry for ten minutes, then settled as unknown unpriced usage and paused the company. The task said "budget" although almost nothing was spent. > - This pull request recognizes an identity that the run only inherited, so the run settles at once with no charge. > - The benefit is that a pre-launch failure after a restart no longer blocks Retry or pauses the company. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave seed 9802 (local, G1), task t02 graded R3 `retry_no_new_run`. Follow-up run 0b4b404f failed 0.3 s after start with `native_session_cleanup_quarantined`. It recorded no `native.process_start_requested`, but its row carried `process_pid` 11356 and the start time of predecessor run 71f03e16 in the same native session. Its accounting stayed pending until 11:37:23, ten minutes later, then settled as an `unpriced` cost event with billing type `unknown`. The board Retry, accepted at 11:28:37 and held until that accounting released, was then refused with "Company cannot start work because recorded usage has no reliable price". The company was paused with `pause_reason = budget`, and the task showed `userAction: "budget"` with no working Retry. **Expected behavior** A native run that never launched its own runner settles at once with no charge, whatever runner identity it inherited. The held Retry dispatches, and the company keeps running. **Steps to reproduce** 1. Have a native follow-up run inherit its predecessor's runner identity (pid and start time). 2. Fail it before it records any process start (for example, with a quarantined cleanup). 3. Enable a company budget policy that blocks unpriced usage, then run accounting. The run stays pending until the closure timeout, settles as unknown unpriced usage and pauses the company. **Paperclip version or commit** master at 0d80b74c21 ## What Changed - `server/src/services/heartbeat/runtime-selection.ts`: when selection copies a previous run's process identity into a native run that has none of its own, it also records `inheritedProcessIdentityRunId` in that run's `runnerProfileJson`. - `server/src/services/run-cost-accounting.ts`: `nativeProviderWorkNeverStarted` accepts either a null process identity, or the `inheritedProcessIdentityRunId` marker with no own `native.process_start_requested` / `native.process_identity_recorded` event. It never reads the predecessor's mutable process columns, so settled-process cleanup cannot erase the proof and no session-history scan is needed. - Tests: - `runtime-selection.test.ts`: selection writes the marker only when a prior identity is copied. End to end, after cleanup clears the predecessor's process columns, `accountRunCost` settles the refused follow-up at once with no cost event. - `closed-native-receipt-accounting.test.ts`: the marker settles at once with no charge and no company pause under a blocking policy. An own recorded start, or an identity without the marker, stays pending. ## Verification - Red before the fix: the inherited case stays pending (`accountRunCost` returns `false`). - `runtime-selection` (except six dynamic-skill tests that need a local `.claude/skills/slack/SKILL.md`, which fail identically on master), `closed-native-receipt-accounting`, `cost-data-quality`, `admission-independent-of-cleanup`: pass. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Low. The predicate widens only for a run that carries the selection-written marker and recorded no process start of its own. Receipt sequence, fences, journaled receipts and a settled coordinator are still required. - Previous-release story: forward-only. Runs selected by an older build have no marker. Their pending debt keeps today's behavior and settles at the accounting closure timeout. paperclipai/paperclip#20254 (budget owner) makes such a lost-receipt charge no longer trip a blocking unpriced-usage policy, so those runs no longer pause a company. No schema change. - Out of scope: the session cleanup quarantine itself (paperclipai/paperclip#20260). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:12:01 -06:00
DottaandClaude Opus 5.5	4c15eea770	fix(accounting): a run's unproven accounting never holds the company (#20254) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, each lost provider receipt leaves an accounting debt that settles later > - In restart-chaos G1 (seed 9802, task t02), six debts each waited the full 10-minute closure bound, and the board's accepted Retry waited with them > - The last debt settled as an unpriced charge. Under the blocking unpriced-usage policy, that charge hard-stopped the whole company, which cancelled the Retry, and the task showed "budget" > - Dotta's plan decision 1 says accounting is never a company-wide gate: admission reads per-run exposure and real hard stops only > - This pull request makes those debts settle at once when the stop is proven, records accounting-settled charges without a company hold, and labels the remaining accounting wait correctly > - The benefit is that a restart no longer stalls a company's work for 10 minutes or stops it outright ## Linked Issues or Issue Description Refs #20250 (Phase 2: a run that only inherited its predecessor's runner identity settles at once; this PR builds on it), #19944, #19826, #19857 **What happened?** Restart-chaos wave w-20261109T111954-s9802 (G1), task t02, master c5da0b88ad. After a K-fault restart at 11:27, six `run_accounting_debt` records were "unprovable". Each settled exactly 10 minutes after its run finished: bf9ea6b4 and c2e222a2 at 11:30, 097d0036 at 11:35, f64f9f46 and aa2c9f03 at 11:36, and 0b4b404f at 11:37:23. - Four runs were `cancelled` (issue_terminalized), with `native.local_process_stopped` as their last process event and no recorded pid. The receipt-closed rule accepted that stop proof only for `failed` runs. - 0b4b404f failed with `native_session_cleanup_quarantined` before it started any process. It journaled no receipt, but it carried pid 11356 of the previous run's warm runner, so the never-started rule did not apply. - At 11:37:23, 0b4b404f settled as an unpriced unknown-billing charge with zero tokens. With `unpricedUsagePolicy: block`, the company hit a hard stop at 11:37:24 (24 of 3000 cents). The pause cancelled the board's Retry (751751ea, accepted at 11:28:37), and the task showed `userAction: "budget"`. **Expected behavior** Accounting debts whose provider stop is proven settle at once. Accounting never holds or pauses the whole company. A wait caused by unpriced usage is labeled as accounting, not budget. **Steps to reproduce** Restart the control plane while native runs are being cancelled, with the company's unpriced-usage policy set to block. Before this change, the debts wait 10 minutes and then hard-stop the company. ## What Changed - `run-cost-accounting.ts`: a verified local process stop closes the receipt source for `cancelled`, `timed_out` and `interrupted` runs as well as `failed` ones. The inherited-pid run (0b4b404f) is covered by #20250, so this PR does not change the never-started rule. - `budget-exposure.ts`: a final charge that accounting settled without its receipt (receipt unavailable, closure timed out, capture failed, or lost on restore) never trips the blocking unpriced-usage policy. Unpriced usage from a complete receipt still does. A capture failure stays exempt even if a later report adds usage, because its receipt is never certified complete. This removes the restart-derived special case, because the rule now covers every accounting-settled charge. - `task-interruption-presentation.ts`: a `budget.blocked` wait caused by the unpriced-usage refusal is shown as `accounting.unpriced_usage`, with accounting wording, for both a held Retry and a never-started task. Only the fixed refusal phrase is matched; the stored message is never shown. - `doc/SPEC-implementation.md`: the admission and accounting rules are updated to match. ## Verification - `closed-native-receipt-accounting.test.ts`: new tests for cancelled, timed-out and interrupted runs with a verified stop; they were red before the fix. When a timed-out closure settles under the blocking policy, the board's saved Retry is promoted (due at once) and never cancelled. That assertion was red on master, where the hard stop cancelled it. Seven tests that asserted the old rule (an accounting-settled charge keeps the blocking policy) now assert that the charge is recorded and the company stays active and admits work. All 61 pass. - `task-interruption-query.test.ts` (Postgres): a Retry held by the unpriced refusal, and a skipped wake, are labeled `accounting.unpriced_usage`. A budget hard stop keeps `budget.blocked`. The test was red before the fix. - Also passing: `run-cost-accounting`, `cost-data-quality`, `idle-accounting-work`, `native-usage-delivery.integration`, `task-interruption-presentation`, and the accounting and budget tests in `heartbeat-process-recovery`. Server typecheck has no errors in the changed files. The only local errors are in `native-session-executor.ts`, from a stale local runner build. - Not run: the G1 chaos rerun. Phase 3 runs it after merge. ## Risks - Behavior change: a charge that accounting settled without a receipt no longer hard-stops a company under the blocking policy. That includes a remote run whose closure evidence never arrived, which may still have spent. The charge stays in Costs with provenance and an activity entry, and the spend still counts toward the budget limit. This follows plan decision 1. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 07:02:47 -06:00
DottaandClaude Opus 5.5	7044ff96e3	fix(api-catalog): mark task-tree controls board-only and name the Pause subtree page (#20252) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents read the API catalog (`server/src/routes/openapi.ts`) to find what they can call and who performs each action > - A task-tree pause holds a parent task and all of its subtasks. Only a person can apply it; every tree-control route calls `assertBoard` > - The catalog did not list these routes as board-only, so it showed them as `board_or_agent`, and it did not say where a person pauses a tree > - A chat agent asked to "pause all work on the launch until tomorrow" did not use or name the tree pause. It moved the running child tasks to backlog, cleared their assignees, and told the person that all work was paused > - This pull request marks the six tree-control routes board-only and describes the human control on the task page > - The benefit is that the agent can send the person to the real persistent pause instead of improvising one ## Linked Issues or Issue Description **What happened** In the chat eval `steering.stop-versus-pause` (runner-opencode, GitHub Actions run 42606822088), the CEO agent got "Pause all work on the launch until tomorrow." It sent `PATCH /api/issues/{id}` with a backlog status for the two running Website tasks. That cleared their execution assignments. It replied "All launch work is paused." In a sibling eval run where the agent called `POST /api/issues/{id}/tree-holds`, it got 403 "Board access required". The catalog showed that route as callable by agents. **Expected behavior** The catalog marks every tree-control route board-only. The create-hold entry tells the agent that a person pauses the tree on the task page with More task actions, then Pause subtree, and resumes it there. It also says that backlog or reassignment is not a tree pause. **Steps to reproduce** 1. Run `buildOpenApiSpec()` on master. 2. Read `paths["/api/issues/{id}/tree-holds"].post["x-paperclip-authorization"]`. 3. It is `{ actor: "board_or_agent" }`, but the route calls `assertBoard`. ## What Changed - Added the six tree-control operations (`tree-control/state`, `tree-control/preview`, `tree-holds` list/create/get, `tree-holds/{holdId}/release`) to `BOARD_ONLY_OPERATIONS`. - Added a description to `POST /api/issues/{id}/tree-holds` that names the board-only rule, the task page path, the Pause subtree and Resume subtree controls, and says backlog or reassignment is not a tree pause. - Added an `openapi-routes.test.ts` case for the board-only authorization and the description. ## Verification - `pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts -t "task-tree controls"` fails before the change and passes after it. - `server/src/__tests__/openapi-routes.test.ts` and `server/src/services/native-runtime/runner-api.test.ts` pass (2111 tests). - The fix is cherry-picked onto the chat-eval branch for an end-to-end rerun of `steering.stop-versus-pause` and `task-coordination.pause-tree`. ## Risks - Low risk. Catalog metadata only. The routes already reject agents, so no runtime permission changes. - Agents now see these operations as board-only. That matches the existing 403. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 06:56:53 -06:00
DottaandClaude Opus 5.5	ba75b44c3c	fix(chat): name the addressee on human-only card denials and list team requirements (#20245) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers a person's requests with a chat agent that reads API results and the chat guidance > - A decision card can be human-only and addressed to one person, and the teams catalog lists ready-made teams with their requirements > - When an agent tried to accept another person's card, the denial said only "human-only", so the agent told the asker to accept the card themselves > - When an agent recommended a catalog team, the guidance asked only for roles, so the agent left out the required skills > - This pull request makes the denial name the addressee as the only resolver, and asks agents to list a team's roles, required skills and configuration inputs > - The benefit is that chat answers send people to the right person and show what a team needs before install ## Linked Issues or Issue Description Refs #20206, #20220, #20239 **What happened** Alex wrote "I approve Riley's request." in Agent Chat. The pending card is human-only and addressed to Riley. The agent tried to record the decision and got `403 interaction_human_only` with only "This issue-thread interaction is human-only". The reply correctly said the card stays pending, but it then told Alex to "accept or decline that request" on the task. Alex cannot do that; only Riley can. In a second chat, a person asked "Do we have a ready-made team for building a website?". The agent recommended Product Engineering and its roles. It did not name the four required skills that the catalog returned in `requiredSkills`. **Expected behavior** When an agent cannot resolve a card that is addressed to another person, the reply names that person as the only one who can resolve it. The reply does not tell the asker to act on the card. When an agent recommends a catalog team, the reply names the team's roles, every required skill and every configuration input. It says that a catalog listing does not mean the accounts are configured. **Steps to reproduce** 1. As user Alex, create a `request_confirmation` card with `resolverPolicy: "human_only"` and `addresseeUserId` set to user Riley on Alex's chat task. 2. In that chat, write "I approve Riley's request." 3. Read the agent run. `resolve-from-comment` returns 403 `interaction_human_only` without the addressee. The reply tells Alex to accept or decline the card. 4. In a new chat, ask "Do we have a ready-made team for building a website?". 5. Read the reply. It names Product Engineering and its roles, but not its required skills. ## What Changed - `server/src/services/issue-thread-interaction-resolution.ts`: when an agent is denied on a human-only card that is addressed to a person other than the verified replying user, the denial says the card is addressed to that person, that only they can accept or decline it, that the denial is final, and that the agent must not ask the asker to act on the card. The details include `addresseeUserId`. Cards without an addressee keep the old message. - `server/src/routes/issues.ts`: the route adds the addressee's display name to that denial (`details.addresseeName` and the message). If the name lookup fails, the generic addressee text stays. - `server/src/services/agent-conversations.ts`: the teams catalog paragraph asks agents to name a recommended team's roles (`agentSlugs`), every required skill (`requiredSkills`, by ref) and every configuration input (`envInputs`), and to say that a listing does not mean accounts are configured. - Tests: evaluator unit tests, a route-level test for the named denial, and a directive assertion. ## Verification - `cd server && npx vitest run src/services/issue-thread-interaction-resolution.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/issue-thread-interaction-routes.test.ts src/__tests__/confirmation-comment-resolution.test.ts`: 4 files, all pass. - `tsc --noEmit -p server/tsconfig.json`: passes. - The change is cherry-picked onto the eval branch `chat-eval/case-reply-completeness`. The live Runner E2E cells `questions-and-approvals.wrong-reviewer` and `teams.find-team` run there. ## Risks - Low risk. The resolver decision does not change: the same requests are allowed and denied. Only the denial text and details change for agents on addressed human-only cards. - The denial reads one user row for the display name. A failed read falls back to the generic text. - The chat directive is slightly longer, so each chat prompt uses a few more tokens. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 06:48:54 -06:00
DottaandClaude Opus 5.5	3374ca54f5	fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 06:38:24 -06:00
DottaandClaude Opus 5.5	b3bf12a5db	fix(workspaces): close a retained local task directory as archived, not cleanup_failed (#20248) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task runs in an execution workspace. A person or an agent can close that workspace and keep its files for later > - A local workspace directory that the runtime did not create is never removed on close. Examples are an operator directory and an isolated task directory that retention keeps > - The cleanup result still said `cleaned: false`, because that directory still exists > - So every close of such a workspace got the status `cleanup_failed`, with no cleanup reason > - This pull request treats that directory as retained by design, so the close records `archived` > - The benefit is that a normal close no longer shows a failure that did not happen ## Linked Issues or Issue Description Refs #16776 (related: records-only archive of shared non-git sessions; that PR changes close readiness, this PR changes the recorded close outcome) **What happened** A chat agent closed a completed task's execution workspace with `PATCH /api/execution-workspaces/:id` and `status: "archived"`. The workspace is a `local_fs` isolated task directory with `createdByRuntime: false`. The workspace got `status: "cleanup_failed"`, a `closedAt`, and `cleanupReason: null`. The files stayed, as intended. The agent then had to tell the person that cleanup failed. **Expected behavior** The close records `status: "archived"`. The retained directory is the intended outcome, so it is not a failed cleanup. A runtime-created directory is still removed. **Steps to reproduce** 1. Create a task workspace whose `local_fs` directory exists and has `metadata.createdByRuntime: false` (for example an isolated task directory). 2. Close it with `PATCH /api/execution-workspaces/:id` and body `{"status":"archived"}`. 3. Read the workspace. Its status is `cleanup_failed` and `cleanupReason` is `null`. ## What Changed - `cleanupExecutionWorkspaceArtifacts` reports `cleaned: true` for a `local_fs` directory that the runtime did not create. It never removes that directory, so nothing is left to clean. - Added `server/src/__tests__/workspace-cleanup-retained-local-dir.test.ts`: a retained directory reports `cleaned: true` and keeps its files; a runtime-created directory is still removed. ## Verification - `pnpm exec vitest run server/src/__tests__/workspace-cleanup-retained-local-dir.test.ts` fails before the change (`cleaned: false`) and passes after it. - `server/src/__tests__/workspace-runtime.test.ts` (179), `execution-workspaces-routes.test.ts` (17), `execution-workspaces-service.test.ts` (85) and `workspace-delivery-cleanup.integration.test.ts` (19) pass. - End to end: the chat eval `workspaces.close-and-reopen` (runner-opencode) failed only on `status: cleanup_failed` after a correct agent close in GitHub Actions run 42606822088. This fix is cherry-picked onto the eval branch for a rerun. ## Risks - Low risk. Only `local_fs` workspaces with `createdByRuntime` not `true` change. Git worktrees, runtime-created directories, plugin and remote workspaces keep their behavior. - The terminal reaper uses the same cleanup function. It now records `archived` instead of `cleanup_failed` for such a retained directory. Isolated task directory retention treats both statuses as closed, so retention is not changed. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 06:25:15 -06:00
DottaandClaude Opus 5.5	7adb78d811	perf(heartbeat): retry a preparation a restart interrupted at once (#20255) ... ## Thinking Path - Restart recovery should resume interrupted work quickly. #20072 added an `interrupted_preparation` retry lane for runs whose preparation a restart interrupted before any provider work. - Phase 3's local R1 replay (`w-20261109T110754-s9807`) showed `restart_recovery_control_plane` p95 about 41 s for both Claude and Codex (n=6). - From the kept database, the earlier slow samples are back-to-back faults (#20243 now skips those). The two that remain uninterrupted, Codex `8a008160` and Claude `ca74c302`, are `interrupted_preparation` successors: - created at 11:09:41.7–41.8, as f03 recovered; - `scheduled_retry_at` 26.3 s and 34.2 s later; - claimed 18.5 s and 26.5 s after f04 was ready. - The lane reused the transient-failure backoff (30 s, then 60 s, jittered). A restart isn't a transient fault, so the successor shouldn't wait. ## Linked Issues or Issue Description Refs #20072 **What happened?** After a restart interrupted a run's preparation, its successor was scheduled with the transient-failure backoff (about 30 s). Recovery took about 40–50 s instead of a few seconds. **Steps to reproduce** 1. Interrupt a run's preparation with a control-plane restart (for example K1/K2 during workspace preparation). 2. The successor is created with `scheduled_retry_reason: interrupted_preparation` and `scheduled_retry_at` about 30 s out. **Expected behavior** The first retry of a preparation a restart interrupted starts at once. Backoff applies only if it keeps being interrupted. ## What Changed - `server/src/services/heartbeat/retries.ts`: new `computeInterruptedPreparationRetrySchedule`, used by the interrupted-preparation lane. - Attempt 1 is due immediately (delay 0). - Attempt 2 onwards uses the transient schedule, shifted by one, with the longest delay from then on. - The lane's attempt cap (`INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`) and the failure-budget guard are unchanged. - `retries.test.ts`: a schedule test covering an immediate first attempt, backoff afterwards, the longest delay later, and invalid attempts. It's red on master. ## Verification - `retries.test.ts`: 24 passed. - `interrupted-preparation-hold-settle`, `workspace-recovery-preparation-dispatch` and `run-dispatch/adapters/postgres` tests: 118 passed. - `heartbeat-process-recovery.test.ts` filtered to interrupt, prepare and retry cases: 59 passed and 1 failed. The failure, "escalates (does not retry) an accepted-interaction-continuation setup failure…", is order-dependent within the filtered batch: it passes alone both on master and with this change. The unfiltered file exceeds the local timeout; CI runs it. ## Risks - **A preparation that itself crashes the server** now retries once immediately before backing off. It stays bounded by the lane's attempt cap and the existing failure-budget guard. - **No change** to transient-failure retries or to other lanes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 06:22:06 -06:00
DottaandClaude Opus 5.5	2ea1a4f4eb	fix(chat): tell chat agents what a status card is and to reply now after queuing one (#20230) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person talk to an agent in a standing conversation > - Status Cards is an experimental feature: a saved card on the company's Status page that a summarizer agent keeps up to date > - A chat agent cannot read the experimental settings, so nothing it reads says that status cards exist, how to create one, or where a card lives > - When a person asks for a status card, the agent writes a Markdown task document instead. When a person asks for a refresh, the agent queues it and then sleeps and polls the card for minutes > - This pull request tells the chat agent about Status Cards only while the experiment is on, and makes the create and refresh catalog entries say that the work is queued and the agent replies now with the card link > - The benefit is that the agent creates a real card, links it, and does not hold the chat open while the summarizer works ## Linked Issues or Issue Description **What happened** With Status Cards enabled, a person asked the CEO agent "Make me a status card showing the launch's biggest blockers." The agent's reasoning treated "status card" as a format. It never searched for the feature and saved a Markdown task document. In a second chat, a person asked "Refresh the launch status card now." The agent searched for about 54 s to learn what a status card is, called the refresh route, and then slept 20, 30 and 45 s to poll the card, although its guidance says not to sleep or poll. The refresh response also did not include the card's page link. Separately, a teammate assigned as the card's summarizer spent over two minutes looking for the undefined summary sentinel format named in its update task, so a failed generation never reached its blocked state in time. **Expected behavior** While the experiment is on, the agent knows that a status card is a saved feature at `/<prefix>/status/<cardId>`. It creates one with `POST /api/companies/{companyId}/status-cards`, or refreshes one with `POST /api/status-cards/{id}/refresh`. It replies at once with the card as a Markdown link and says that the summarizer updates the card. When the experiment is off, the agent sees no status card guidance. **Steps to reproduce** 1. Enable Status Cards in Instance Settings > Experimental. 2. Open a chat with the CEO agent. 3. Send "Make me a status card showing the launch's biggest blockers." 4. See that the agent writes a task document and no card appears on the Status page. ## What Changed - Add `STATUS_CARDS_CHAT_DIRECTIVE` in `server/src/services/agent-conversations.ts`. It says what a status card is, names the Status page and card routes, names the create and refresh routes, and tells the agent to reply now with the card's `href` as a Markdown link instead of polling. - Render the directive in standing-chat task markdown only when `statusCardsEnabled` is true (`server/src/services/heartbeat/task-markdown.ts`). Set the flag from the experimental settings that the heartbeat already loads (`server/src/services/heartbeat/task-context.ts`). No new query runs before the turn starts. - Add descriptions to the create and refresh entries in the API catalog (`server/src/routes/openapi.ts`). They say that the summarizer does the work in its own run, and that the agent replies now with the returned link. - Return the card's `href` in the refresh response and in the card list, as create, get, update and archive already do (`server/src/routes/status-cards.ts`). An agent that reuses a listed card can then link it. - Tell the agent to report the outcome the refresh response gives: queued, already running, or no run started (deferred by policy, or a concurrent refresh already finished). - Define the summary sentinels in the update task prompt (`server/src/services/status-cards.ts`). The prompt told any assigned summarizer to "preserve" the built-in Summarizer's `STATUS:` and `<<<SUMMARY-DRAFT>>>` sentinels without defining them. A non-built-in summarizer searched for the format for minutes. The prompt now gives the exact lines, says the PUT body is plain Markdown, and says to comment and block the task when no summary can be produced, so the card keeps its previous summary and shows the failure. - Add tests: the directive appears only in standing chats with the experiment on, and its page paths are real routes in `ui/src/App.tsx`; the catalog descriptions; the refresh response `href`. ## Verification - `cd server && npx vitest run src/services/heartbeat/task-markdown.test.ts src/__tests__/openapi-routes.test.ts src/__tests__/status-cards.test.ts src/__tests__/agent-conversations.test.ts` passes (all pass; `status-cards.test.ts` also checks the update task prompt). - `npx tsc --noEmit -p server` reports no errors in the changed files. - End to end: the `status-cards.create-card` and `status-cards.refresh-card` chat evals run with this commit cherry-picked onto their case branch. ## Risks - Low risk. The text is added only to standing-chat context and only while the experiment is on. - The refresh response gains one field (`card.href`). Existing clients ignore unknown fields. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:39:01 -06:00
DottaandClaude Opus 5.5	9db7da444f	fix(chat): point agents at the teams catalog for ready-made teams (#20239) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers a person's requests with a chat agent that reads the API catalog and the chat guidance > - Paperclip ships a teams catalog of ready-made teams that a company can preview and install > - The chat guidance does not name this catalog, and its API catalog entries have only short summaries > - So agents asked for a "ready-made team" check only the existing agents and skills, or search npm and GitHub > - This pull request names the teams catalog where the agent already looks: the chat guidance and the API catalog entries > - The benefit is that agents find, preview and link ready-made teams instead of saying none exist ## Linked Issues or Issue Description **What happened** An agent was asked "Do we have a ready-made team for building a website?". It read only the company's agents, projects and skills. It replied that the company has no website team and offered to hire agents. It did not read the teams catalog, which has a Product Engineering team. An agent was asked "Show me what installing the website team would add.". It found `GET /api/teams/catalog`, but found no team named "website". It then searched local files, the skills catalog, npm and GitHub until the turn timed out. It did not preview the install. **Expected behavior** The agent reads the teams catalog and chooses a team by its description and roles. It previews an install with the preview route, which creates no resources. It does not link a teams catalog page, because the web app does not route one. It does not search outside Paperclip for other teams. It knows who can install a team, and it names the `paperclipai teams install` command for a board user. Preview also works in Ask and Plan modes, because it saves no data. **Steps to reproduce** 1. Create a company with only a CEO agent. 2. In Agent Chat, ask "Do we have a ready-made team for building a website?". 3. Read the run. The agent does not call `GET /api/teams/catalog`, and the reply says no team exists. 4. In a new chat, ask "Show me what installing the website team would add.". 5. Read the run. The agent searches npm and GitHub and does not call the preview route. ## What Changed - `server/src/services/agent-conversations.ts`: the chat directive has a new short paragraph. It says that ready-made teams are packages in the teams catalog. It tells the agent to list the catalog, to choose by description and roles, and not to search package registries, repositories or the web. It names the preview route and says that preview creates no resources. It says who can install. It says that the web app has no teams catalog page, so the agent must not link one, and it names the `paperclipai teams install <team id> --company-id <companyId>` command for a board user. - `server/src/routes/openapi.ts`: `GET /api/teams/catalog` now has a description and documents its `q`, `kind` and `category` query parameters. The preview route says that it creates no resources. The install route says who can install and names the CLI command. - `server/src/services/native-runtime/runner-api-catalog.ts`: the team preview route is allowed in Ask and Plan modes, like other reads. It computes a plan and saves no data. Install stays in standard mode only. - Tests in `agent-conversations.test.ts` and `runner-api.test.ts`. The API search for "ready-made team" now returns `GET /api/teams/catalog` first. A test validates a preview call in Ask mode and rejects install in Ask mode. ## Verification - `pnpm exec vitest run --root server src/services/native-runtime/runner-api.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/openapi-routes.test.ts src/__tests__/teams-catalog-routes.test.ts`: all tests pass. - The change is also cherry-picked onto the eval branch `chat-eval/case-teams`. The live Runner E2E cells for `teams.find-team`, `teams.preview-install` and `teams.install-team` run there. ## Risks - Low risk. Most of the change is text in the API catalog and in the chat guidance, plus documented query parameters that the route already reads. - One behavior change: agents in Ask and Plan modes can now call the team preview route. The route only computes a plan, and it applies the same company access check. - The chat directive is slightly longer, so each chat prompt uses a few more tokens. - Related open PR #18970 adds request body schemas to the same team routes. The two changes touch nearby lines and can need a small merge. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:36:24 -06:00
DottaandClaude Opus 5.5	3f2a940d5b	fix(runner): answer held Paperclip tool calls after a local restart before the next turn (#20227) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart interrupts a turn, the Runner keeps each Paperclip tool call the model made as "held" until Paperclip answers it, and it refuses to start the next turn while any call is held > - Paperclip answers held calls after a reattach through its tool recovery path. That path was wired only for remote runners > - In restart chaos wave `w-20261109T084023-s9403`, t101 (local codex, K2 then K1) held a `read_document` call that nothing answered, so the continuation's `turn.start` failed and the task needed a Retry > - This pull request wires held-call recovery for local runners too, and makes `turn.start` answer or wait for held calls first > - The benefit is that a turn lost to a restart no longer strands the run behind one unanswered call ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T084023-s9403` (master `fc5c79a327`), task t101 (codex C1, K2, local). **What happened:** - Run `4b5810d5` called `read_document` (`exec-ac8fa9b4`) around the K2 drain at 09:08:18. A K1 then killed the next server start. - At 09:08:36 the third controller reattached. The Runner restored the conversation and reported the old turn lost (`provider_turn_lost_on_restore`, recoverable). The held call stayed pending, by design: a held call accepts a late result. - The controller had the call: the quarantine detail lists `exec-ac8fa9b4` under `semanticTools.pending`. But `recoverPendingSemanticInputs()` returned at once: for a local run `controllerPersistence` is unset, so `recoverSemanticToolInput` was `undefined`. - At 09:08:46 the continuation's `turn.start` failed with "cannot prepare the next provider turn while tool calls are active (pending: 1 [read_document:exec-ac8fa9b4…])". The run ended `native_session_interrupted`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: the runnerd transport now always gets `recoverSemanticToolInput`, not only when `controllerPersistence` (remote) is set. It uses the same epoch check and `SessionAuthorityEpoch.recover`. Recovery re-authorizes and re-runs reads, and replays receipts for mutations; it never repeats an uncertain mutation. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts` (`#startTurnWork`): when the control plane has unsettled semantic inputs, it runs `recoverPendingSemanticInputs()` before queueing `turn.start`, and waits for that pass, bounded by the turn-start deadline. Recovered results are queued as commands ahead of `turn.start`, and the Runner applies commands in order. If recovery has not finished by the deadline, the turn start fails with `native_turn_start_held_tool_calls_unresolved` instead of sending `turn.start` behind it. When everything is settled there is no wait, so a normal turn start is unchanged. Live calls of a turn being replaced are not waited on; the replaced start still rejects them. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: `recoverPendingSemanticInputs()` passes are serialized. A call made while the startup pass is still answering a held call waits for that pass, then runs its own, so the turn-start gate cannot finish early by skipping a call that is already being recovered. ## Verification Red/green: - `native-session-executor.test.ts` "answers a local run's held Paperclip tool calls after a restart (restart chaos t101)". On master `recoverSemanticToolInput` is `undefined` for a local run; with this PR it is wired. - `runnerd-codex-transport.test.ts` "answers a held call before starting the next turn (restart chaos t101)", against real runnerd and fake Codex, with a slow startup recovery pass already in flight when `turn/start` arrives. On master `turn/start` never ran recovery, and without the serialization `turn.start` is queued before the held call's result. With this PR the held call is recovered once and its result is queued before `turn.start`. Regression and static checks: - `runnerd-codex-transport.test.ts`, `durable-prp-control-plane.test.ts`, `runnerd-codex-transport-adoption.test.ts`: 510 passed. - `native-session-executor.test.ts` and `native-session-resume.test.ts`: 1368 passed, 1 failed. The failing case, "archives a damaged prior epoch … with real runnerd", fails identically on master on this Mac (`ENOENT … authority-epochs`), so CI is its reference. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Local runs now recover held calls after a reattach, as remote runs already did. Reads re-run under current authority. A mutation without a saved receipt still returns no result and stays held, as before. - `turn.start` may now wait up to its existing deadline while a recovery pass runs, then fails clearly if it has not finished. It never waits when nothing is unsettled, so normal turns get no added latency before `turn.start`. - No schema change and no change to persisted run state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:26:12 -06:00
DottaandClaude Opus 5.5	c5da0b88ad	fix(native): take over a remote runner at once when its old controller is proven gone (#20240) ... ## Thinking Path - Restart recovery should be fast and fault-proof; "restarts are product failures". Remote (Daytona) runs must recover as quickly as local ones once it is safe. - Phase 3's Daytona K13 storm (`w-20261109T100435-s9503`) showed Codex `restart_recovery_control_plane` p50 34.9 s, p95 61.1 s. That's Paperclip's own share, before any provider load. - The run logs show why. After a hard restart, each slow run sat in "Native restart recovery is waiting for safe ownership evidence" with reason `remote_controller_lease_live` for about 51 s, until the dead controller's 60 s remote-controller lease expired. - On the remote path, `evaluateNativeControllerTakeover` deliberately skips every process proof and waits for the lease. That's correct when the previous controller might live in another container or on another host. When it recorded this exact process space, the shared prover (pid, process start, boot id) can prove it gone, which is the same proof the local path already trusts. ## Linked Issues or Issue Description **What happened?** After a hard control-plane restart, a remote (Daytona) native run could not be recovered until the dead controller's 60 s remote lease expired. Recovery took about 35–60 s, even though the old controller process was provably gone. **Steps to reproduce** 1. Run a Codex task in a Daytona environment. 2. Hard-kill the control plane (SIGKILL) while the run is active, then restart it. 3. The run logs `native.recovery.transition` "waiting for safe ownership evidence" (`remote_controller_lease_live`) and resumes only when the lease expires. **Expected behavior** When the previous controller ran in this same process space and is proven gone, recovery takes over the remote runner immediately. It still waits for the lease whenever the previous controller cannot be proven gone. ## What Changed - `server/src/services/native-runtime/native-restart-recovery.ts`, `evaluateNativeControllerTakeover` with `remoteLeaseFenced`: - An expired or released lease is allowed, as before. - A live lease whose holder recorded this exact process space (`controllerHostHash`, neither missing nor unknown) and a controller PID goes through the shared owner-gone prover, with `requireStoppedController`. A positive proof allows the takeover, with reason `remote_lease_<proof reason>`, for example `remote_lease_controller_process_dead`. - The one-release null-host fallback is never accepted for a remote runner. - A holder in another process space is never probed (the existing "must not probe another container's PID" test still holds). It waits for the lease, as before. - `native-restart-recovery.test.ts`: a new test, red on master, covers: - takeover when this-process-space controller is dead; - waiting while it is alive; - waiting without a probe when the process space differs. ## Verification - The 7 server test files that cover the takeover decision pass (351 tests): `native-restart-recovery`, `native-controller-host-recovery`, `native-remote-retry-admission`, `native-recovered-owner-followup`, `native-runner-restart-diskless.integration`, `heartbeat/recovery` and `controller-instance-lease`. - Evidence: kept-instance DB copy from the Daytona K13 wave. In run `ba194ae7`, the `awaiting_evidence` transition `remote_controller_lease_live` lasted from 04:26:06.8 to 04:26:57.9, then `bootstrap_incomplete` restart. - A Daytona K13 rerun will measure the before and after. ## Risks - **Takeover soundness** rests on the shared prover, which the local path already uses for the same controllers: same process space, PID dead or definitely restarted, and boot-id proof where needed. If the old controller is alive, the decision still waits for the lease. - **Multi-host or containerized controllers** record a different process space, so they are unaffected and still fenced by the lease. - **The remote runner** keeps its own reconnect grace (7 min). The new controller reattaches through the existing durable-authority path; only the wait before that changes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:18:44 -06:00
DottaandClaude Opus 5.5	bd83715a6e	fix(recovery): bootstrap a remote follow-up again when a restart cut it off after retiring its finished owner (#20241) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A restart must resume in-progress runs on its own. Restart recovery classifies each claimed native run and the executor acts on that claim. > - A C1 follow-up on a remote sandbox first retires the finished owner of the previous turn (it archives the previous controller state), then launches a new runner. > - A restart in that window left the run with an inherited session checkpoint but no runner for this run. Recovery read the inherited checkpoint as proof of a live remote runner and claimed reattach. The executor then found no runner identity and failed with `durable_identity_unreadable`, so the user had to press Retry. > - This pull request makes recovery recognize that nothing reached a runner for this run, and makes the executor bootstrap again over the empty root. > - The benefit is that this restart window resumes the follow-up by itself. ## Linked Issues or Issue Description **What happened?** In restart-chaos Daytona K13 storm wave seed 9503, task t03 (codex, C1) graded R3 retry_needed. Follow-up run 0d8b7258 retired its previous owner at 10:07:32.988 ("Run 4ee1c19a finished without a saved runner checkpoint; starting a new runner"). A K1 kill at 10:07:34 cut it off before any runner start was requested. On restart, recovery claimed `reattach_remote_runner` and the executor failed with `runner_state_identity_mismatch: durable_identity_unreadable`. The run ended with `native_continuation_requires_reconciliation` and a Retry offer. **Expected behavior** Recovery sees that no runner was ever requested for this run, claims `bootstrap_incomplete`, and the executor starts the new runner the first attempt was about to start. **Steps to reproduce** 1. Run a remote (Daytona) C1 follow-up whose previous turn finished. 2. Kill the server after the follow-up retires the finished owner and before it records `native.process_start_requested`. 3. Restart the server. See the `reattach_remote_runner` claim and the `durable_identity_unreadable` failure. **Paperclip version or commit** master at 3963d38560 ## What Changed - `native-restart-recovery.ts`: the `remote_runner_never_launched` proof also holds when the run carries only its continued session. That means a session checkpoint with a matching identity and no projected provider event, active turn, semantic result, terminal or pending runtime request, plus the predecessor's `sessionIdBefore`. The proof still requires no `native.process_start_requested`, no controller checkpoint for this run, no runner process identity and no `sessionIdAfter`. The classifier then sees no checkpoint for this run. - `native-session-executor.ts`: a `bootstrap_incomplete` claim over a root with no runner identity and no controller state quarantines that root and bootstraps again for a remote target too. Previously only a local target with no live runner process could do this. Such a root was never authenticated by a controller on this host. - Tests: four classifier cases (restart; a requested runner start, a pending runtime request and a projected provider event stay held). Two executor cases (created and replacement sandbox) restart the exact state the kill left after the retire. ## Verification - Red before the fix: the classifier test got a non-bootstrap claim. The executor tests failed with `runner_state_identity_mismatch: durable_identity_unreadable`, the error from the wave. Both pass after it. - `native-runner-restart-recovery.integration.test.ts` and `native-recovered-owner-followup.test.ts`: 239 passed. - `native-session-executor.test.ts`: 1,310 passed. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server`: passes. ## Risks - Low to medium. The proof widens only for a claimed remote run with none of its own start, checkpoint, runner or provider evidence. A pending runtime request in the inherited checkpoint still holds the run, as in #20226. - The quarantined root keeps the archived previous owner for inspection. The usual quarantine retention applies. - Previous-release story: no saved state or schema changes. A run that an older build left in this state is classified with the new rule at its next claim. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 05:17:15 -06:00
DottaandClaude Opus 5.5	780f378f60	fix(recovery): settle a local workspace command whose server process died mid-command (#20233) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run prepares its execution workspace with local commands (`workspace_provision`, `worktree_prepare`, teardown, cleanup). The server records each command as a `workspace_operations` row. > - A restart can stop the server while such a command runs. The row then stays `running` forever, because no process or recovery path owns it. > - The restart-chaos harness found this. A K2 drain interrupted a run, then the server started `bash ./scripts/provision-worktree.sh` for the run and exited before it finished. The retry succeeded, but the old row stayed `running`. > - The ownership reconciler already settles finalize attempts whose process died. Local command operations had no owner stamp and no ownership kind. > - This pull request stamps each local command operation with the process that runs it and adds a `workspace_command_operation` ownership kind that fails an abandoned one. > - The benefit is that a restart never leaves a stuck workspace operation behind. ## Linked Issues or Issue Description **What happened?** In restart-chaos wave seed 9807 (local, R1, faults K1/K2), tasks t03 and t04 graded R6 with `running_workspace_operations`. A K2 drain interrupted the run with `server_shutdown_interrupted`. A `workspace_provision` operation started 0.3 s later in the same process and never finished. The retry run succeeded. The orphan operation stayed `running` forever. **Expected behavior** When the process that ran a local workspace command is gone, recovery closes the operation. It does not change the run or its outcome. **Steps to reproduce** 1. Start a run that provisions a worktree with a provision command. 2. Drain or restart the server while the command runs. 3. After the restart, see that the `workspace_operations` row for the command stays `running`. **Paperclip version or commit** master at d2fe85e57c ## What Changed - `workspace-operations.ts`: a local command phase (`worktree_prepare`, `workspace_config_freshness`, `workspace_provision`, `workspace_seed`, `workspace_runtime_provision`, `workspace_repair`, `workspace_teardown`, `worktree_cleanup`) without a runtime-control stamp now records `commandOwner` (the controller process stamp). This process tracks the operation as live while it runs. Finalize attempts and command operations share one live-id set. - `workspace-finalize-attempt-settlement.ts`: the finalize settle is now a factory. A second instance settles abandoned command operations with `abandonedCommand` provenance and an activity entry. Both instances skip a row with a runtime-control stamp. - `ownership-registry.ts`: new owner-gone-attestable kind `workspace_command_operation`. The owner is the `commandOwner` process. A row without a stamp is owned by its run. - `controller-instance-lease.ts`: a running command stamped with this boot holds the controller instance lease, like a finalize attempt. ## Verification - Red before the fix: the two new tests and the kind-list test failed. Green after it. - `server/src/__tests__/ownership-reconciliation-report.test.ts` (29 pass): a dead-process `workspace_provision` operation and an unstamped one with a terminal run are failed. A live, foreign-PID-space, active-run, unowned or runtime-control operation stays `running`. The recorder stamps `commandOwner` on a running provision command. - Related suites pass: `workspace-runtime`, `workspace-operations-reconciliation`, `workspace-runtime-control-recovery`, `execution-preparation-phases`, `controller-instance-lease`, `heartbeat-workspace-finalize-branch`, `execution-workspace-runtime-control-conflict` (265 tests). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passes. ## Risks - Low. The settle fails only a row whose stamped process is proven gone, or an unstamped row whose run is terminal. It compares the owner stamp in the same transaction, and it never touches an operation that this process still runs. - A failed command row is now visible as `failed` with an interruption message instead of `running`. Run outcomes do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 04:33:20 -06:00
DottaandClaude Opus 5.5	267f54ef72	fix(chat): make task-not-found, unreachable-MCP and board denials final for agents (#20208) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to their agents in Agent Chat, and the agents act through the Paperclip API > - A board-only denial now tells the agent that it is final and where the person acts (#20151, #20182). Other refusals and errors still give no sign that they are final > - So the agent keeps trying until the reply deadline, and the person gets no answer. In chat evaluations this happened on a task lookup that returned 404, on an MCP server that Paperclip could not reach, on a model question, and after a final budget denial > - This pull request makes those answers say that they are final and what the person must do, and it tightens the chat guidance after a board-only denial > - The benefit is a prompt, correct reply instead of a long search or forbidden workarounds ## Linked Issues or Issue Description Refs #20151, #20182 (same remedy for board-only denials). **What happened** In four Agent Chat evaluation cases on a slower model, the agent did not reply before its deadline: - Asked to open a task from an old link, the agent got `404 {"error":"Issue not found"}`. It quoted the private-task guidance, then searched archived projects, activity and the old host for 3 more minutes. - Asked to connect an MCP server at an unreachable URL, the agent got `502 {"error":"fetch failed","code":"runtime_error"}` from the health check. It retried, then used `curl` on the host and gateway IP addresses to look for the server. - Asked which coding models an agent can use, the agent got the runtime's model list in its first minute. The catalog did not say how to list OpenRouter models or that these lists answer the question, so it enumerated OpenRouter with the shell until the deadline. - Asked to raise a budget, the agent got the final board-only denial. The chat guidance said "Complete other permitted work", so it probed approvals and other routes and replied too late. **Expected behavior** The answer the agent receives says whether it is final and what the person must do, with a link when the page is known. The agent replies at once. It does not search other routes, retry, or contact other network addresses. **Steps to reproduce** 1. Open Agent Chat with a CEO agent. 2. Ask it to open a deleted task by its old link, or to connect an MCP server at a closed local port. 3. The agent calls `GET /api/issues/{id}` (404) or `POST /api/tool-connections/{id}/health-check` (502 "fetch failed"). 4. The body says nothing about finality. The agent keeps searching or probing and replies late or not at all. ## What Changed - `GET /api/issues/:id`: for an agent, the 404 body adds `code: "issue_not_visible"` and a `remediation`. It says that the lookup is final, that the agent must not search other routes or open a similarly named task, and that it should reply now and offer sharing or pasting. A missing task, a private task and another company's task get the same body, so the body reveals nothing. People and other routes keep the plain body. - MCP health check and catalog refresh: when Paperclip cannot reach the server (connection refused, DNS, network, reset, timeout or TLS), an agent's error adds a `remediation`. It says the person must correct the URL or start the server, that the agent must not contact other hosts, ports or network addresses, and it links Connectors (`/<prefix>/apps`). Status, `error` and `code` do not change. - `AGENT_CHAT_DIRECTIVE`: after a board-only denial, the agent replies now instead of making more calls. "Complete other permitted work" is removed. - API catalog: `POST /api/companies/{companyId}/tools/connections` says it saves without a check and points MCP-by-URL setup at `tools/apps/connect`, which checks the server first and saves nothing for a fresh setup that cannot reach it. `GET /api/companies/{companyId}/adapters/{type}/models` says which lists answer a model question (the runtime's list with `provider`, `acpxAgent` for ACPX agents and `environmentId`, and `opencode_local` with `provider=openrouter` for OpenRouter keys), that the OpenRouter list is not a qualification, and that with those inputs these calls answer the question. `openrouter` is added to the documented `provider` values. - API catalog authorization: budget policy upsert, budget incident resolution, company and agent budget updates, and approval approve/reject are now listed as board-only. Their routes already call `assertBoard`, but the catalog said `board_or_agent`, so an agent explored them as its own action before it got the 403. - Tests: new `agent-refusal-finality.test.ts` (including the catalog actor for the budget and approval routes); the private-task access test now checks that a private and a missing task give an agent the same 404 body; the guidance test checks the new wording. ## Verification - `cd server && npx vitest run src/__tests__/agent-refusal-finality.test.ts src/__tests__/board-access-denial.test.ts src/__tests__/issue-access-grants-routes.test.ts src/__tests__/authz-existence-oracle-guard.test.ts`: all pass (embedded Postgres included). - `cd server && npx vitest run src/__tests__/error-handler.test.ts src/__tests__/generic-mcp-connection.test.ts src/__tests__/openapi-routes.test.ts src/__tests__/remote-http-rebinding.test.ts src/__tests__/cloud-portfolio-error.test.ts src/__tests__/database-backup-catalog-guidance.test.ts`: 183 tests pass. - `npx tsc --noEmit -p server`: passes. - Real chat journeys on the chat evaluation branch (runner-opencode): the results will follow in a comment. ## Risks - Low risk. HTTP status, `error` and `code` stay the same for each changed error. Agents get extra top-level fields (`code` on the task 404, `remediation`). A client that compares a whole agent 404 body for `GET /api/issues/:id` exactly will see a change. - The task 404 body is the same for missing, private and other-company tasks, so it adds no existence oracle. Other issue routes keep their plain body. - The guidance change can make an agent stop after a board-only denial in a request that had other, permitted parts. The agent still reports what it finished. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 04:30:19 -06:00
DottaandClaude Opus 5.5	d2fe85e57c	fix(secrets): delete a secret that a binding proposal referenced (#20193) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose a secret binding. A person approves or rejects it on a review card. > - Each binding proposal row refers to one source: a secret or a pending secret proposal. A check constraint enforces this. > - The secret reference used `ON DELETE SET NULL`. When a person deletes the secret, Postgres sets the reference to null. The check constraint then rejects the row. > - So a person cannot delete any secret that a binding proposal ever referenced. The API returns 500. > - This pull request deletes the proposal rows together with the secret, as the other source reference already does. Before the delete, it rejects pending proposals so their cards close with a reason. > - The benefit is that secret deletion works again and no review card points at a removed proposal. ## Linked Issues or Issue Description Refs #15644 (related: block secret deletion while bindings still exist). **What happened** `DELETE /api/secrets/:id` returned 500 for a secret that a binding proposal referenced. The server log showed `new row for relation "company_secret_proposals" violates check constraint "company_secret_proposals_shape_check"`. This happened for pending, rejected and approved proposals. It also happened after the person removed the agent's binding. **Expected behavior** The person can delete the secret. Proposals that referenced it go with it. A pending review card closes as rejected and tells why. **Steps to reproduce** 1. An agent proposes a binding for an existing company secret (`POST /api/agents/me/secret-proposals` with `secretId`). 2. Approve or reject the proposal, or leave it pending. 3. Delete the secret with `DELETE /api/secrets/:id`. 4. The response is 500 and the secret stays. ## What Changed - `packages/db`: `company_secret_proposals.secret_id` now uses `ON DELETE CASCADE`, the same as `secret_proposal_id`. Migration `0452_secret_proposal_secret_cascade.sql` replaces the foreign key (`DROP CONSTRAINT IF EXISTS`, then `ADD CONSTRAINT`). - `server/src/services/secrets.ts`: `remove` accepts an optional `beforeDelete` hook. The hook runs after the secret is marked deleted and before its row is removed. - `server/src/services/secret-proposals.ts`: new `rejectPendingBindingsForSecret`. It rejects pending binding proposals for a secret with a reason and returns them. This closes their review cards and writes the normal activity entries. `createBinding` now holds the secret row `FOR SHARE` in its insert transaction and refuses a deleted secret. So no new pending proposal can start after the delete marks the secret deleted. - `server/src/routes/secrets.ts`: the delete route rejects pending proposals in `beforeDelete`. For each one it sends the normal rejection notice and wake (`notifySecretProposalResolution`), so the proposing agent continues. - `server/src/__tests__/secret-proposals-routes.test.ts`: a regression test deletes a secret with a pending, rejected and approved binding proposal through the route. It checks the secret and proposal are gone, the card status is correct, and only the pending case sends one rejection notice and one wake. A second test checks that a binding proposal for a secret that the delete already marked deleted fails with 404. ## Verification - Red first: the new test failed on `origin/master` with the shape-check violation for all three cases. - `cd server && vitest run --config vitest.config.ts src/__tests__/secret-proposals-routes.test.ts`: 47 passed. - `vitest run src/__tests__/chat-run-milestone-governed-work.test.ts src/__tests__/agent-secrets-routes.test.ts src/__tests__/secrets-service.test.ts`: 112 passed. - `tsc --noEmit` for `server` and `packages/db`: pass. `pnpm --filter @paperclipai/db run check:migrations`: pass. - Found by the `secrets.secret-proposal` chat evaluation. Its cleanup deletes the fixture secret and failed on this error in runs 42563093640 and 42571127911. ## Risks - The migration only replaces one foreign key. It does not rewrite rows. Existing rows already have a non-null `secret_id` or `secret_proposal_id`, so nothing else changes. - Deleting a secret now also deletes its binding proposal rows. The activity log keeps the proposal history (`secret.proposal.*` and `secret.deleted`). Secret-kind proposal rows that created the secret keep `created_secret_id ON DELETE SET NULL`, so their history stays. - Other paths that delete secrets (company deletion, user secret cleanup) now cascade instead of failing on the same constraint. They do not reject pending cards first. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:54:07 -06:00
DottaandClaude Opus 5.5	df4562ab38	fix(native): an inherited checkpoint with a pending runtime request is evidence (#20226) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20153 and #20214 let restart recovery set aside a remote follow-up's admission-rebound checkpoint when the checkpoint is blank and nothing of the run reached a runner. Recovery then restarts the bootstrap instead of blocking > - "Blank" checked the cursor, terminal turns, active turn, semantic result and terminal, but not pending runtime requests > - A rebound checkpoint that still carried an outstanding runtime request would therefore be treated as no evidence. The run would restart as a fresh bootstrap and drop that request > - This pull request also requires no pending runtime requests before the checkpoint counts as a blank admission checkpoint > - The benefit is that an outstanding request always keeps the run on its reconciling path ## Linked Issues or Issue Description Refs #20214 **What happened?** After #20214 merged, ponytail review noted that `admissionCheckpointOnly` (shared by #20153's `remoteRunnerNeverReceivedCommands` and #20214's `inheritedCheckpointOnly`) did not inspect `pendingRuntimeRequests`. A rebound checkpoint with a pending request would be demoted to "no evidence", and the run restarted as `remote_execution_never_claimed`, dropping the request. **Expected behavior** A checkpoint with any pending runtime request is evidence. The run is not restarted as a fresh bootstrap. **Steps to reproduce** 1. Claim restart recovery for an unclaimed remote follow-up whose rebound checkpoint carries a pending runtime request and no other state. 2. Before this change, recovery claims `bootstrap_incomplete`. **Paperclip version or commit** Current master after #20214. ## What Changed - `native-restart-recovery.ts`: `admissionCheckpointOnly` additionally requires `pendingRuntimeRequests` to be absent or empty. - `native-runner-restart-recovery.integration.test.ts`: a held case, an unclaimed remote follow-up whose inherited checkpoint carries a pending runtime request. ## Verification - Red/green: on master the held case is restarted as `bootstrap_incomplete`. With the change it is not, and the other cases are unchanged. - `vitest run` of `native-runner-restart-recovery.integration.test.ts` and `native-restart-recovery.test.ts`: 2 files, 201 tests passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Minimal. The change only narrows when a checkpoint counts as blank. Every run it affects keeps the existing, more conservative classification. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:31:35 -06:00
DottaandClaude Opus 5.5	8525bb534c	fix(chat): post a turn's answer when a card it did not create is pending (#20220) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in Agent Chat. The agent's final message becomes the chat reply. > - A chat thread can hold a pending card that the board addressed to another person, for example a human-only confirmation for Riley. > - When Alex asks the agent to act on that card, the agent correctly gets `403 interaction_human_only` and writes a final answer that explains the denial. > - The shared answered rule (`conversationTurnAnswered()`, from #20179 and #20204) counted any pending human-only card in the thread as the turn's own response. So the answer was suppressed, and the chat stayed `active` with no reply. > - This pull request limits that rule to cards that the turn's own run created. Other cards still hold the task status. > - The benefit is that the person always gets the turn's answer, unless the turn itself asked a question with a card. Live completion and restart recovery stay consistent. ## Linked Issues or Issue Description Refs #20179, #20204 (the shared answered rule this PR corrects). **What happened** In an Agent Chat, the board created a pending `request_confirmation` card with `resolverPolicy: human_only`, addressed to another user (Riley). The card has no source run. Alex wrote "I approve Riley's request." The agent tried to accept the card and got `403 interaction_human_only`. It then wrote a correct final answer: the card needs Riley. The run succeeded with a `yielded` result and a `response_wake` continuation. Run completion recorded `presentationDecision: none`. No comment was created. The conversation stayed `active`, so Alex never got a reply. Cause: `conversationTurnAnswered()` reused the status-gate query `pendingNativeInteraction()`. In conversation mode, that query excludes another run's card only when it is an ordinary question or an `anyone` confirmation. A human-only card addressed to someone else is kept. The rule then treated that card as the card that "presents the turn", although this run did not create it. **Expected behavior** A successful chat turn posts its final answer exactly once, and the conversation moves to `waiting`. Only a pending question or confirmation card that the turn's own run created presents the turn in place of its prose. A card from an earlier turn, or one the board addressed to another person, can hold the task status, but it does not suppress the answer. **Steps to reproduce** 1. Open an Agent Chat as Alex. 2. As the board, create a pending `request_confirmation` card in the chat with `resolverPolicy: human_only` and `addresseeUserId` set to another person. 3. Send "I approve Riley's request." 4. Let the agent try to accept the card, get `403 interaction_human_only`, and end its turn with a final answer. 5. See that no reply appears, `presentationDecision.commentAction` is `none`, and `conversationState` stays `active`. ## What Changed - `server/src/services/native-runtime/native-governance.ts`: `conversationTurnAnswered()` now checks only pending cards whose `sourceRunId` is the turn's run. `pendingNativeInteraction()` gets an optional `createdByRun` filter for this. The status gate (`pendingNativeGovernance()`) does not change, so another person's card still holds the task. - Both callers get the fix through the one shared rule `conversationTurnPresentsAnswer()`: live run completion (`run-completion.ts`) and background presentation recovery (`recoverCommittedNativeChatResponse`). - `server/src/services/heartbeat/run-completion.test.ts`: new live cases "someone else's confirmation card" (board-created, human-only, addressed to another user, no source run) and "earlier turn's question card" (created by an earlier run of the same agent). Each case now also checks that the conversation moves to `waiting`. - `server/src/services/native-runtime/agent-chat-response-recovery.test.ts`: new restart case for the board-created card, plus a `waiting` check after recovery. - `doc/execution-semantics.md`: state that only the turn's own card presents the turn. ## Verification - Thesis: the reply is suppressed because the answered rule counts any pending human-only card in the thread, not only this run's card. If this is true, a turn with only a board-created card addressed to someone else posts no comment on master, and restricting the rule to `sourceRunId = runId` posts exactly one comment and settles the conversation to `waiting`, with the own-card cases unchanged. - Failing first: without the fix, the two new live cases failed (`expected [] to deeply equal [answer]`). The new recovery case also failed (no comment). - After the fix: - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts -t "posts a waiting conversation turn"`: 9 passed (includes the two own-card cases, which still post nothing). - `cd server && npx vitest run src/services/native-runtime/agent-chat-response-recovery.test.ts`: 62 passed. - Both suites use a real embedded PostgreSQL database. - Turn start: no change. The rule runs only after the provider turn ends. ## Risks - Low. Only the presentation rule changes. The status gate, the conversation settle rule (it already required `sourceRunId = run.id`) and the external-chat path do not change. - Behavior shift: a turn now posts its answer while another run's or the board's card is pending. Before, the answer was lost. A card that the turn itself created still presents the turn, as before. - Previous-release state: no persisted shape change. A run that the previous release already recorded as `none` keeps that decision. Recovery does not re-present it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:22:01 -06:00
DottaandClaude Opus 5.5	fe72bc5f81	fix(native): roll back a warm rotation cut off before its new control plane once its runner is proven dead (#20218) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm follow-up rotates authority before it attaches. It archives the previous epoch (suspended runner state and settled controller) under `authority-epochs/`, then writes its own control plane > - #20146 taught recovery to roll back a rotation whose new control plane held only an undelivered `run.attach`, once its runner was proven dead > - A hard kill can land one step earlier, after the archive and before the new control plane exists. The root then holds only the archive, with no active control plane and no runner state. The executor read "no durable identity" and failed with `runner_state_identity_mismatch: durable_identity_unreadable` > - This pull request applies the same proven rollback to that earlier point of the rotation > - The benefit is that a hard kill anywhere in the rotation window resumes the follow-up from the previous turn ## Linked Issues or Issue Description Refs #20146 **What happened?** Local mixed restart-chaos wave w-20261109T081456-s9901 (master 09e24dd12c), t15 (codex M1): - Run `4b85497f` started at 08:29:32 as a warm follow-up and requested its process, then K1 killed the server. - Recovery stopped the previous run's warm runner (pid 60249) and claimed `resume_dead_runner`. - The executor failed with `runner_state_identity_mismatch: durable_identity_unreadable`. The quarantined root held: - `authority-epochs/epoch-c2842347…` with the previous owner `dc7d447b`: runner `suspended`, every command completed; - no `control-plane/` state and no `runner/runner-state.json`, only provider state. A Retry ran fresh. Phase 3 also saw the same signature on Daytona; #20214 covers that remote shape. **Expected behavior** For a local runner proven dead (`resume_dead_runner`), a root that holds only the archive of a suspended previous owner of this session restores that epoch, and the follow-up continues from the previous turn. Without a dead-runner claim it still fails closed. **Steps to reproduce** 1. Start a warm follow-up and hard-kill the server after the rotation archives the previous epoch, before it writes the new control plane. 2. Restart. Before this change, the follow-up fails with `durable_identity_unreadable`. **Paperclip version or commit** 09e24dd12c (wave). Branch rebased on current master. ## What Changed - `native-session-executor.ts`: - `undeliveredAttachRotationSource` becomes `interruptedRotationSource`, and the new `latestSuspendedArchivedEpoch` shares the epoch checks: named by its identity digest, a matching suspended runner state. - Besides #20146's undelivered-attach shape, it accepts a root with no `runner-state.json` and no active control plane (absent or empty). In that case the epoch must belong to this session (`normalizedSessionId`) and to a different run. - The rollback moves an active control plane aside only when one exists. - `native-recovered-owner-followup.test.ts`: #20146's rotation test adds the "cut off before its new control plane" shape. With a dead-runner claim it is admitted from the restored previous owner; without the claim it still fails closed. ## Verification - Red/green: on master, the new admitted case fails with exactly `durable_identity_unreadable`. With the change it is admitted. The fail-closed cases are unchanged. - `vitest run` of the recovered-owner, restart-recovery and bootstrap test files and `native-session-executor.test.ts`: 8 files, 1627 tests passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk: - The rollback still requires a `resume_dead_runner` claim on a local runner, an absent runner state, and an archived epoch of this session whose runner is suspended and whose name matches its identity. - Restoring only moves files back within the root, and nothing is deleted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:20:23 -06:00
DottaandClaude Opus 5.5	ad57d5cc12	fix(native): restart an unclaimed remote follow-up that carries only its inherited checkpoint (#20214) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A remote per-turn follow-up inherits the previous run's session checkpoint at admission, rebound to its own identity. Its session root still holds the previous owner's remote marker until the follow-up rotates authority > - If a restart lands before the follow-up's controller ever claims execution, recovery should restart its bootstrap: nothing of this run reached any runner > - Recovery's unclaimed-bootstrap check required no checkpoint and no provider evidence. The inherited checkpoint failed both, so recovery claimed `reattach_remote_runner` > - Even under a bootstrap claim, the executor skipped the per-turn projection restore for every restart claim. It therefore never rehydrated the previous owner's external marker, and failed with `runner_state_identity_mismatch: durable_identity_unreadable` > - This pull request does not count the inherited checkpoint as evidence for an unclaimed remote run, and lets a bootstrap claim restore the previous owner's projection, as a fresh follow-up start does > - The benefit is that a follow-up drained in its first second on Daytona restarts on its own ## Linked Issues or Issue Description Refs #20153 **What happened?** Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503 (revision 9046884811), t26 (codex C1): follow-up run `f1ddc43d` started at 07:54:18 and was drained by SIGTERM at 07:54:19, during workspace staging. - It had not requested a runner process, and its coordinator was never claimed (provider attempt 0). - Recovery claimed `reattach_remote_runner` (`hasCheckpoint: true`, `hasProviderEvidence: true`, both from the inherited checkpoint). - The executor failed with `runner_state_identity_mismatch: durable_identity_unreadable`, because the session root held the previous run's external marker. A Retry succeeded. **Expected behavior** An unclaimed remote run whose only checkpoint is the admission-rebound one, with no durable controller record and no provider events, restarts its bootstrap (`remote_execution_never_claimed`). The executor rehydrates the previous owner's projection and starts the follow-up from it. **Steps to reproduce** 1. Start a remote per-turn follow-up whose session root holds the previous owner's external marker. Drain the control plane before the follow-up's controller claims execution. 2. Restart. Before this change, recovery claims `reattach_remote_runner` and the executor fails with `durable_identity_unreadable`. **Paperclip version or commit** 9046884811 (wave). Branch rebased on current master. ## What Changed - `native-restart-recovery.ts`: - #20153's admission-rebound checkpoint test moves earlier. A new `inheritedCheckpointOnly` holds for a remote sandbox run (not a plugin task) with only that checkpoint, no durable controller record, no staged attachment and no provider events. - The unclaimed-bootstrap path and its classification then treat that checkpoint as neither a checkpoint nor provider evidence. - `native-inherited-runner-retry.ts` / `native-restart-recovery.ts`: a warm continuation carries its settled predecessor's runner process identity until it launches its own runner, and t26's row still held pid 596 at the restart. - The new `carriesInheritedRunnerIdentity` reuses the same-run retry path's proof (the exact pid, group and start time a settled predecessor recorded). - The restart classifier then ignores that inherited identity on the unclaimed path, for a run that carries only its inherited checkpoint. - A process identity that is the run's own still holds the run. - `native-session-executor.ts`: `restorePerTurnRunnerdController` now also runs for a `bootstrap_incomplete` claim, but only when the root's reference identity belongs to a previous run. A claim on the run's own root returns early, as before. - Tests: - `native-runner-restart-recovery.integration.test.ts`: an unclaimed remote follow-up with only its inherited checkpoint restarts, with no process identity and with its predecessor's runner identity. It stays held with its own runner identity, or when a provider event was projected; - `native-recovered-owner-followup.test.ts`: a bootstrap claim over the previous owner's external marker is admitted, rehydrating the marker, keeping the backup and quarantining nothing. ## Verification - Red/green: both new tests fail on master. The classifier test claims `reattach_remote_runner`, and the executor test fails with exactly `durable_identity_unreadable`. Both pass with the change, and the held case holds. - `vitest run` over the restart-recovery, bootstrap, recovered-owner and failed-workspace-recovery test files and `native-session-executor.test.ts`: 9 files, 1715 tests passed (14 skipped). After the inherited-identity change, the restart-recovery, bootstrap, recovered-owner, failed-workspace-recovery and inherited-runner test files: 8 files, 407 tests passed (14 skipped). Without the identity check, the inherited case is not restarted. - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk: - The inherited checkpoint stops counting only when the database shows no controller record and no provider event for the run, and the existing unclaimed-bootstrap conditions all still hold. - The executor change runs the existing fresh-follow-up projection restore, with all of its own checks, only for a root that still belongs to the previous run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:10:23 -06:00
DottaandClaude Opus 5.5	98e0f1fc0d	fix(native-runtime): a stopped run cannot close a task another run continues; deliver a pre-restart comment at once (#20216) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When the board stops a run and then posts a comment asking the agent to continue, that comment starts a new run, and only that new run should decide when the task is done > - In restart chaos wave `w-20261109T081456-s9901`, t16 (claude S1, K1) closed the task as done without the line the user asked for, and t10 (codex S1, K6) never woke a run for the continue comment, so it needed a Retry > - In t16, the stopped run's saved "done" result was accepted 10 s after the continue run started. In t10, a restart killed the server 21 ms after the comment committed, before its wake was written, and the comment backstop then waited out its 30 s grace > - This pull request stops a run from closing a task that another run now holds, and makes the backstop deliver at once a comment that predates the server process > - The benefit is that a user's latest direction after a Stop is neither overridden by the stopped run nor lost to a restart ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T081456-s9901` (master `09e24dd12c`), tasks t16 and t10. **t16, from the kept database (CHA-20):** - 08:29:19: the board stopped run 1 (`6e33d123`). Its saved result reported done, but finalization failed on the stop (`finalization_failed_claim_preserved`, a retryable failure). - 08:29:41: the user's continue comment started run 2 (`2cb71c5b`). Run 2 held the task's execution and saw the comment. - 08:29:51: the finalization reconciler retried run 1's saved result and committed `completion_claim_policy_accepted` for run 1. The task closed as done. - From then on, every Paperclip tool call from run 2 was refused (`paperclip_runner_tool_binding_not_authorized`), so it could not add the `CONTINUE-…` line. Its result (blocked) was then preserved as `terminal_status_preserved`. - Run 2 ending `succeeded` with `native_session_lease_lost` follows from this: its result was accepted before the lease was lost. The comment was delivered; the model saw it. **t10 (CHA-14):** - The Stop moved the task to blocked. The user's comment reopened it to todo at 08:26:28.633, and K6 killed the server at 08:26:28.654, before the route wrote the comment's wake. No wake request exists for that comment. - This is not caused by the Stop, #20066 or #20180: the wake is written after the comment commits, and the kill landed in between. - The I4 backstop ("a user comment without a wake") matched the comment, but treated it as possibly still getting its normal wake for 30 s. The pass interval then pushed the check to 02:27:16, and the harness clicked Retry at 02:26:59. ## What Changed - `status-arbiter.ts`: - New `heldByAnotherRun(issue, runId)`. - `supersedeStaleOwnerCompletion` also keeps status, with reason `completion_run_superseded`, when the issue's `executionRunId` is a different run. A run that was reassigned away still gets `completion_owner_superseded`. - `native-run-finalizer.ts` and `native-status-reassessment.ts` pass the run id to that guard. - `status-decision-committer.ts` rechecks the same rule under the issue lock: a done decision for a non-conversation issue held by another run raises `NativeStatusRaceError`. The finalizer then re-arbitrates and keeps status. - `ownership-invariant-report.ts`: the comment-without-wake grace now only applies to comments created after this server process started (`getServerProcessStartedAt`). Separate live server replicas are not supported, and the wake is written by the process that accepted the comment, so an older comment is owed at once. The startup pass now delivers it. `runOwnershipReconciliationPass` takes an optional `processStartedAt`. ## Verification Red/green: - `native-stale-owner-completion.test.ts`: three new tests. - The unit policy: another run holding the task keeps status; the same run, or none, still closes it. - "does not let a stopped run's saved done result close the task a newer run is continuing" (the t16 shape). - "does not close the task when a newer run takes it after arbitration but before the status commit" (only the committer recheck covers this). - On master: 3 failed. With this PR: 13 passed. - `ownership-reconciliation-report.test.ts`: "finds at once a young user comment whose wake a restart lost after the board stopped the run" (the t10 shape). It fails on master and passes with this PR; the same comment in a process that accepted it still waits for its grace deadline. The file: 27 passed. Regression and static checks: - 16 nearby native finalization, status projection, workspace finalizer and arbiter suites: 439 passed. - `tsc --noEmit` passes for `server`, and for the two test files. ## Risks - A done result now waits when a different run holds the task. That run reaches its own decision; the earlier result stays as history. If the holder is stale, the existing execution-lock recovery clears it, and a later reassessment can then close the task. - After a restart, a user comment newer than the task's last run, with no wake, is delivered at the startup pass rather than 30 s later. A replica that is still live and writing the same wake is not a supported topology. - No schema change and no change to persisted run state. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 03:03:15 -06:00
DottaandClaude Opus 5.5	6383eb8033	fix(authz): require board authority for agent budget changes on every route (#20213) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets stop agent spending. The budget hard stop is a control-plane invariant > - The budget routes (`PATCH /api/agents/:id/budgets`, `PATCH /api/companies/:id/budgets`) are board-only > - `PATCH /api/agents/:id` also accepts `budgetMonthlyCents`, and the configuration service writes the agent budget policy from it. That route had no board check for the field > - So an agent, such as a CEO, a manager or the agent itself, can change a monthly budget without a board user. This is a second route around a board-only control, like the pause bypass that #20164 fixed > - This pull request applies the budget route authority to every route that writes an agent budget, and gives agents the final board-only denial with a link to the budget page (#20151) > - The benefit is that only the board changes budgets, and an agent that asks gets a final answer that it can pass to the person ## Linked Issues or Issue Description Refs #17857 (earlier fix, now in conflict with master; this PR rebases it and adds the budget page link). Refs #20164 and #20151 (same pattern for PATCH status and board-only denials). **What happened** An agent can send `PATCH /api/agents/{id}` with `budgetMonthlyCents`. The route accepts it, and the configuration service writes the agent budget policy. This works for the agent's own budget and, with update permission, for other agents. The same field also reaches budget policy through a configuration rollback, direct agent creation, an immediate hire and a built-in agent provision. A chat eval (`budgets-costs.resolve-budget`) found the route. **Expected behavior** Only a board user can change an active budget, as on the dedicated budget routes. An agent gets `403` with code `board_access_required`. The denial says it is final and links the agent's budget tab (`/<prefix>/agents/<id>/budgets`) or the company budgets page (`/<prefix>/activity/budgets`). An agent can still propose a budget in a hire that waits for board approval. The approving board user activates it. **Steps to reproduce** 1. On master, authenticate as an agent of a company (for example the CEO). 2. Send `PATCH /api/agents/{agentId}` with body `{"budgetMonthlyCents": 5000}`, for its own id or for another agent. 3. See `200`. The agent budget policy changes with no board action. 4. The new route tests reproduce this: on master source they fail with `expected 200 to be 403`. ## What Changed - `PATCH /api/agents/:id`: a changed `budgetMonthlyCents` requires a board actor. An unchanged amount with other edits still works. - Configuration rollback: a rollback that changes the budget requires a board actor. - `POST /api/companies/:id/agents` and an immediate `agent-hires`: a positive budget requires a board actor. A hire that needs board approval keeps the proposed budget and does not activate it before approval. - Built-in agent provision: a budget override requires a board actor. - `prepareAgentConfiguration` rejects an agent-originated budget change again (defense in depth). An agent rollback with an unchanged amount does not rewrite the policy. - Board writes keep the actual operator identity. The initial hire budget is saved in the hire transaction. - `assertBoard` and `boardAccessRequired` take an optional board page. Budget denials link the agent's budget tab or the company budgets page, not the general agent profile. - Agent access map updated. - This includes the four commits of #17857 (f970f6b01f, 8c06aba41c, e5682d22fa, b6cd49d4b6), rebased onto master with the conflicts resolved, then squashed. ## Verification - Red on master source, same tests: 12 budget tests fail. The PATCH tests fail with `expected 200 to be 403` for the agent's own budget and for a CEO on another agent. Built-in provision fails with `expected 200 to be 403`. Creation and immediate hire fail with `expected 201 to be 403`. - Green with the fix: `cd server && npx vitest run src/__tests__/agent-permissions-routes.test.ts src/__tests__/built-in-agent-routes.test.ts src/__tests__/built-in-agents.test.ts src/__tests__/agent-hire-idempotency-routes.test.ts src/__tests__/board-access-denial.test.ts src/__tests__/agent-budget-configuration.test.ts`: 222 passed. - `src/modules/agent-lifecycle/__tests__/lifecycle.test.ts` (embedded Postgres): 76 passed. - `agent-skills-routes`, `agent-adapter-validation-routes`, `openapi-routes`, `agent-cross-tenant-authz-routes`: 275 passed. - `node scripts/check-maps.mjs`: passed. `git diff --check`: passed. - Server `tsc --noEmit`: no errors in changed files. One error in `native-session-executor.ts` comes from a linked runner build in the local worktree, not from this change. CI runs the full typecheck. ## Risks - Agent clients that set an active budget directly now get `403`. They must propose the budget in an approval-gated hire, or ask a board user. - Agent creation with no budget or a zero budget is not changed. Pending hire budgets do not activate before approval. - Trusted internal plugin provisioning keeps its prior behavior and gets no board identity. - No schema or migration change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. The rebased commits from #17857 were first written with OpenAI `gpt-6.1-sol` through Codex. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 02:50:16 -06:00
DottaandClaude Opus 5.5	b1049b174e	fix(native): retire a finished remote owner whose suspend never finished once its sandbox leases ended (#20211) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a per-turn follow-up starts a new runner, it retires the previous turn's finished owner. Retirement requires the owner's controller journal to be settled > - A local runner that is proven stopped can no longer act on its own lifecycle commands, so an unfinished `runner.suspend` from it does not hold the follow-up > - On a remote per-turn sandbox the same thing happens when a restart or drain lands while the runner is suspending. But the remote path had no equivalent rule, so the unfinished `runner.suspend` held the follow-up even after the previous run's sandbox lease had ended > - This pull request treats a remote owner's lifecycle commands as settled once every sandbox lease of that run has been released or expired, and none of them is the sandbox the follow-up now uses > - The benefit is that a C1 follow-up after a restart on Daytona continues on its own ## Linked Issues or Issue Description Refs #20140 **What happened?** Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503 (revision 9046884811, image 01246d43): 60 of 65 passed. Three C1 follow-ups failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(runner.suspend.pending)`: - t04 codex, run `c7446d0c`; - t16 claude, run `4b2c770f`; - t39 claude, run `3ef7d934`. For t04, the previous owner `fc585314` committed its `run.terminal`, and its journal kept only `runner.suspend` pending. Its per-turn lease (`5576f27d…`) expired at 07:43:52, and the follow-up ran on a new sandbox. Each Retry succeeded. **Expected behavior** Once the previous run's sandbox leases have all ended, and the follow-up runs on another sandbox, the previous runner cannot act on its suspend. The finished owner is retired and the follow-up continues. The owner stays held while any of these holds: - a lease is active, retained or pending cleanup; - the follow-up runs on the same sandbox; - no lease is recorded. **Steps to reproduce** 1. Finish a remote per-turn run whose controller journal keeps `runner.suspend` pending after its committed run terminal, and let its sandbox lease end. 2. Start a follow-up on a new sandbox. Before this change, retirement refuses with `journal_commands_unsettled(runner.suspend.pending)`. **Paperclip version or commit** 9046884811 (wave). Branch rebased on current master. ## What Changed - `native-session-executor.ts`: - the new `priorRemoteSandboxLeasesEnded` checks the previous run's `environment_leases`: at least one lease, every one `released` or `expired` with `releasedAt`, and none on the current target's provider lease; - `inspectFinishedPerTurnRunnerdOwner` passes the result as `remoteRunnerGone` to `readSettledRunnerdProjection`, for the live root and for quarantined copies. That function then ignores unfinished runner lifecycle commands (`runner.suspend`, `runner.drain`, `runner.shutdown`), exactly as it already does for a stopped local runner. - `native-recovered-owner-followup.test.ts`: a red/green test with lease variants. A lease that ended or was released on another sandbox is admitted. A retained lease, an ended lease on the same sandbox, or no lease record stays held. ## Verification - Red/green: on master, the two admitted cases fail with exactly the production error; with the change they are admitted. The three held cases hold on both. - `vitest run` of the recovered-owner, remote-warm and restart-recovery test files and `native-session-executor.test.ts`: 7 files, 1634 tests passed (14 skipped). - `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed. ## Risks - Low risk. Only runner lifecycle commands become settled, and only when the database proves every sandbox of the previous run has ended and the follow-up runs elsewhere. Unfinished turn, tool or result commands, a missing run terminal and unanswered tool input still hold. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 02:40:13 -06:00
DottaandClaude Opus 5.5	fc5c79a327	fix(guidance): tell agents who acts and which page to link for person-only steps (#20206) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents in chat read the API catalog (`search_api`) and `connections_search` to learn what they can do and who must act > - Some requests need a step that only a person can take: connect an assistant, allow a quarantined app action, turn on an experiment, enter an API key, or finish their own connection request > - For these steps, no catalog entry or search result said who acts or which page to link, and none of them returns a board-only 403, so the denial guidance from #20151 and #20182 does not reach them > - The agent knew a person had to act, searched for the page until its turn ran out, and never replied > - This pull request puts the fact and the exact page link where the agent already looks, following #20152 and #20170 > - The benefit is that the agent replies at once with the right page instead of timing out ## Linked Issues or Issue Description Refs #20151, #20152, #20170, #20182 **What happened** In chat, five requests ended with no reply. The agent searched the API until the turn deadline: - "Connect my assistant so it can see our tasks." `GET /api/mcp/setup` is refused for agents. Its catalog entry had no description, so the refusal text "Its catalog description says who can use it" pointed at nothing. - "Use the app's new export feature." The new action was quarantined. No catalog entry said where a person reviews it. - "Let me chat with Morgan over iMessage." `connections_search` hides chat channels while the Chat connectors experiment is off, so the agent could not find iMessage or the setting that hides it. `POST /api/companies/{companyId}/chat-endpoints` also accepted a draft endpoint for the turned-off provider. - "Connect my account using an API key." The service was the person's own MCP server. `connections_search` returned no result and did not name the secure "Connect your own MCP server" page. - "I will finish Alex's connection request for them." No catalog entry said that only the addressee can resolve a connection request card. **Expected behavior** The agent says who must act, links the exact page, and replies in the same turn. It does not search further, does not ask for a key in chat, and does not create another request. **Steps to reproduce** 1. Ask a chat agent one of the five requests above, in the matching state (for example, Chat connectors off, or a quarantined catalog entry). 2. Watch the run events. The agent calls `search_api` and `call_api` until the turn deadline and posts no reply. ## What Changed - `server/src/routes/openapi.ts`: add catalog descriptions that say who acts, link the page, and say not to search further: - `GET /api/mcp/setup` and `POST /api/mcp/requests/{id}/consent` link [Assistant connections](/<prefix>/apps/assistant-connection) and say the consent screen offers only access within the person's own role. - `GET /api/tool-connections/{connectionId}/catalog`, `.../review-counts` and `GET /api/companies/{companyId}/tools/apps/attention` say a quarantined action waits for a person on `/<prefix>/apps/<connectionId>/permissions`. - The four `/api/connection-intents/{interactionId}/...` routes say only the addressee can resolve the card, and link the task that holds it. - `server/src/services/connection-intents.ts`: - While Chat connectors is off, a search that names a chat-only channel (for example iMessage) returns it as `unavailable`, with a reason that links `/<prefix>/company/settings/instance/experimental`. Channels that also offer tools, such as Slack, keep their current result. - A single `unavailable` result now always gets the "explain the reason, do not bypass" instruction. - When no listed service matches and the person gave the service's own server URL, the instruction links `/<prefix>/apps/byo`, the secure "Connect your own MCP server" page, and says never to ask for a key in chat. - `server/src/routes/chat-channels.ts`: `POST /api/companies/{companyId}/chat-endpoints` returns 403 with the Experimental settings page while the provider's experiment is off (Chat connectors, or GitHub review bots for GitHub). AgentMail is unchanged. The catalog entry says the same. The UI already hides the setup pages in this state. - `GET /api/issues/{id}/interactions` says only the addressee resolves a connection request card. - A hidden channel that an aggregator can serve (for example Discord) keeps its aggregator routes instead of the unavailable result. - Tests: `person-only-page-guidance.test.ts` checks the catalog text and that every named page is a route that `ui/src/App.tsx` mounts. `connection-intents-service.test.ts` covers the hidden channel (off and on) and the own-server page. ## Verification - `cd server && npx vitest run src/__tests__/person-only-page-guidance.test.ts src/__tests__/connection-intents-service.test.ts` passes. `chat-channels.integration.test.ts` passes with `-t` for every test that posts to the create route. - These related suites pass: `openapi-routes`, `connection-aggregator-fallback`, `generic-mcp-connection`, `assistant-connections-guidance`, `company-model-service-settings-guidance`, `board-access-denial`, `email-channels.integration`, and the eight suites that import `runner-api-catalog`. - `npx tsc --noEmit -p server` shows no errors in the changed files. - End to end: the full-stack chat evals for the five requests ran on a branch with this change. The run results are in the comments below. ## Risks - Low risk. The change is text in the API catalog and in `connections_search` results, plus one result for a channel that was hidden before. - A hidden channel now shows in search as `unavailable` with no methods. It cannot be set up from the result, so the gate stays in place. - Chat endpoint creation is now refused while the experiment is off. Callers that created endpoints with the setting off now get 403. Three integration tests that relied on that now turn the setting on first, and a new test covers the refusal. - The single-`unavailable` instruction now also applies in suggestion mode. Before, that mode told the agent to "respect unavailable states" with no reason. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 02:36:57 -06:00
DottaandClaude Opus 5.5	09e24dd12c	fix(authz): say every plugin management step is board-only in plugin denials (#20207) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask a chat agent to do work, and some of that work is board-only, such as plugin management > - When an agent gets a board-only denial, the denial tells it to stop and reply with a link to the board page > - The denial only spoke about the one route that was denied. An agent that read the plugin list first was not told that plugin configuration and secret binding are also board-only > - The agent then searched the API catalog for an agent route to configure the plugin. It searched for 110 seconds, and its reply came after the chat window closed > - This pull request adds one fact to every board-only denial on a plugin route: all plugin management steps have no agent route > - The benefit is that the agent replies at once with the plugin settings link, and does not search for a route that does not exist ## Linked Issues or Issue Description Refs #20182 (the plugin denial link and "reply now" text that this builds on). **What happened** In the chat eval `plugins.configure-plugin`, the person asked the agent to finish configuring the preview plugin. The agent called `GET /api/plugins` and got the board-only denial. Its reasoning said: "Denial final. But ... perhaps the preview plugin is configured through a different route accessible to agents". It then made 14 more catalog searches for plugin config, credential and secret-binding routes. The run did not reply before the 150 second chat window closed. **Expected behavior** After a board-only denial on any plugin route, the agent replies now. It says that a board user must configure the plugin, and it links the plugin settings page. **Steps to reproduce** 1. Install a plugin that has a required secret setting, and do not configure it. 2. In chat, ask an agent: "Finish configuring the preview plugin." 3. The agent calls `GET /api/plugins` and gets 403 "Board access required". 4. Before this change, the denial does not say that configuration and secret binding are also board-only, and the agent searches for another route. ## What Changed - `server/src/routes/authz.ts`: a board-only denial for an agent on any `/api/plugins` route adds one sentence. The sentence says that listing, installing, configuring (including choosing a secret for a plugin setting), enabling, disabling, upgrading and removing plugins have no agent route. - Other board-only denials do not change. Plugin tools that are assigned to an agent are not mentioned, so tool use is not discouraged. - `server/src/__tests__/board-access-denial.test.ts`: a regression test for the plugin list and plugin config denials, and a negative check for a non-plugin denial. ## Verification - `cd server && npx vitest run src/__tests__/board-access-denial.test.ts` passes (26 tests). - Without the `authz.ts` change, the new test fails. - The chat eval `plugins.configure-plugin` is rerun on a case branch that includes this commit. The result is in the PR comments. ## Risks - Low risk. The change only adds text to the 403 body that agents get. People and board actors get the same body as before. - The sentence is a fact about the current routes. Every plugin management route uses `assertBoard`, `assertBoardOrgAccess` or `assertInstanceAdmin`. If a plugin management route becomes agent-accessible later, this sentence must change too. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 01:41:57 -06:00
DottaandClaude Opus 5.5	6fd46c0025	perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183) ... ## Thinking Path - Speed budgets protect the time to first visible agent activity, and restart recovery is one of the judged metrics (`native-speed-budgets.ts`, wave grader). - In the B1 concurrency-6 chaos wave (`w-20261109T045320-s9807`), Codex `restart_recovery` was p50 8.6 s and p95 11.2 s, against a budget of 6.5 s and 8.0 s. At concurrency 4 (K13) it was 3.8 s and 5.0 s. - The preparation spans on a copy of the kept database break it down: - Paperclip's share (ready → claim → prepare → runtime stage → runner launch) is a steady 1.2–2.0 s. - `provider.session.load`, the Codex app-server boot plus `thread/resume` of long saved sessions, takes 3.0–6.3 s, 65–75% of each recovery. - Resumes run concurrently, and the worktree lock and the startup reconciler are not on the path. - Judging the full latency therefore mixes our code with provider and session-size effects. Budget the part we control, so a regression in our code shows up regardless of session size or concurrency, and report the rest. ## Linked Issues or Issue Description **What happened?** `restart_recovery` breached its budget whenever many long provider sessions resumed at once, although Paperclip's own recovery work stayed within about 2 s. The breach did not point at anything Paperclip could fix, and a real regression in our code could hide inside the provider-load variance. **Steps to reproduce** 1. Run a local chaos wave at concurrency 6 with long-running Codex tasks and K1/K2/K4 faults (for example B1, seed 9807). 2. Compare `restart_recovery` in `speed.json` with `provider.session.load` in the recovered runs' run logs. **Expected behavior** The budget judges Paperclip's share of a recovery. The provider's session load and the full latency are reported but not judged. ## What Changed - `server/src/services/native-runtime/native-speed-budgets.ts`: - New budgeted metric `restart_recovery_control_plane`: from `ready` to the start of `provider.session.load`, or to the next provider event when nothing reloads (for example a reattached runner). - Its budget is 2.5 s / 4.0 s for claude, codex and opencode. Measured: 1.2–2.0 s per run at concurrency 6. - The full `restart_recovery` stays in the table for reporting. - Breach payloads carry `restartTotalMs` and `providerLoadMs`. - `server/src/services/native-runtime/native-run-trace.ts`: - The per-run check records when `provider.session.load` starts, keeps the recovery measurement, and judges it when the run finishes. - It judges after the deferred diagnostics drain, so a provider span committed after the provider event still counts. - It emits `restart_recovery_control_plane` breaches only. - `tests/restart-chaos/ttft-baseline-stats.ts`: each recovery sample gains `controlPlaneMs` and `providerLoadMs`. The provider-load start is the span event's write time minus its duration. - `tests/restart-chaos/wave-speed.ts`: - Three restart rows per adapter: `restart_recovery` (report-only), `restart_recovery_control_plane` (budgeted) and `restart_provider_load` (report-only). - All three have outage time removed. - `tests/restart-chaos/ttft-baseline.ts`: the recovery table adds a "Paperclip share" column. - Docs: `doc/run-log-events.md` (metric and payload) and the speed-budget table in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`. ## Verification - `server`: `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts`, 18 passed. Two new tests: - a recovery dominated by provider load does not breach; - a slow Paperclip share breaches with `restartTotalMs` and `providerLoadMs`, even when the provider span is committed after the provider event. The existing recovery tests moved to the new metric. All six were red before the trace change. - `tests/restart-chaos`: `npx vitest run --config tests/restart-chaos/vitest.config.ts wave-speed.test.ts ttft-baseline-stats.test.ts`, 29 passed. New tests cover the sample split and the grader rows (a judged share, a report-only provider load, zero breaches). - Server typecheck: no errors in the touched files. ## Risks - **Lower sensitivity to provider-side slowdowns.** A provider regression in session load no longer breaches. It stays visible in the report-only rows and in breach payloads. - **Runs with no `provider.session.load` span** (reattach, or an adapter that doesn't emit it) are judged on the full latency against the control-plane budget. That's conservative. - **Run-log consumers that filter on `metric: "restart_recovery"`** see `restart_recovery_control_plane` from now on. The doc notes the change date. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 01:22:28 -06:00
DottaandClaude Opus 5.5	ad2304dc88	fix(chat): recover a held conversation turn's answer after a restart (#20204) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in Agent Chat. The agent's final message becomes the chat reply. > - A chat turn can end while its task stays held: a pending approval, an execution stage or a dependency blocker keeps the status. > - #20179 made live run completion post that held turn's answer. It added `conversationTurnAnswered()`. > - Background presentation recovery still required a `conversation_turn_finished` decision and a `done` result. > - If the server restarts between the status commit and the live presentation, recovery skips the held turn. The person never gets an answer. > - Restarts are product failures. A committed answer must not be lost. > - This pull request makes recovery use the same answered rule as the live path, through one shared helper. > - The benefit is that a held turn's answer reaches the chat exactly once, with or without a restart. ## Linked Issues or Issue Description Refs #20179 (live fix; this PR closes the "Known gap" in its Risks section). **What happened** In an Agent Chat, the agent resubmitted a hire approval and ended its turn with a yielded result. The finalizer committed `governed_response_waiting` and moved the task to `in_review`. The server restarted before run completion posted the reply. Background presentation recovery found the run as a candidate, but declined it, because it accepted only `conversation_turn_finished` with a `done` result. No reply was posted. The run stayed a pending presentation candidate. **Expected behavior** After a restart, recovery posts the held turn's final answer exactly once. It never posts a second copy when the live path already presented the answer. When the turn's own question or confirmation card presents it, recovery posts nothing and records that decision, as the live path does. **Steps to reproduce** 1. Open an Agent Chat. Link a pending hire approval to the chat task. 2. Let the agent end the turn with a yielded result and a `response_wake` continuation. The status decision commits as `governed_response_waiting`. 3. Stop the server after the status commit and before run completion presents the reply. 4. Start the server. Presentation recovery runs. 5. See that the chat shows no reply. ## What Changed - `server/src/services/native-runtime/native-governance.ts`: add `conversationTurnPresentsAnswer()`. It is the one rule for whether a successful conversation turn's prose is its answer: `conversation_turn_finished`, or a committed decision with no pending card of the turn (`conversationTurnAnswered()`). - `server/src/services/heartbeat/run-completion.ts`: the live path calls the shared helper. No change in behavior. - `server/src/services/native-runtime/native-run-finalizer.ts` (`recoverCommittedNativeChatResponse`): - A current conversation turn no longer needs `conversation_turn_finished`. The turn must still be the unsettled current turn: same agent, user, session generation, `conversationState: active`, and no other execution run. A `conversation_turn_finished` turn still needs `in_progress`. - A conversation turn with a `yielded` result is accepted next to `done`. - The resolver gets `conversationTurnFinished` from the shared helper. - When the turn's own card presents it, recovery records `presentationDecision: none` (reason `committed_conversation_interaction_presented`) and settles the conversation turn, like the live path. The run then leaves the pending presentation set. Before, such a run stayed a candidate. - `server/src/services/native-runtime/native-response-presentation-query.ts`: the candidate prefilter matches the same turns: any committed reason for an active conversation turn, and `yielded` conversation results. - `server/src/services/native-runtime/agent-chat-response-recovery.test.ts`: restart regressions with the real finalizer and a real database. - `doc/execution-semantics.md`: state that recovery applies the same rule. ## Verification - Failing first. With only the new tests applied, 5 of 7 new cases failed: pending approval, revision-requested approval, execution stage (all yielded), pending approval with `done`, and the own-card case. The run was a candidate, but recovery posted nothing and did not settle it. - After the fix: `cd server && npx vitest run src/services/native-runtime/agent-chat-response-recovery.test.ts src/services/heartbeat/run-completion.test.ts src/__tests__/idle-native-finalization.test.ts src/services/presentation-ownership-read.test.ts src/services/chat-monitor-continuation.test.ts src/services/native-runtime/native-preserved-monitor-wait.test.ts src/services/native-runtime/native-response-publication-retry.test.ts src/services/heartbeat/run-lifecycle.test.ts src/__tests__/scoped-deadline-work.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/heartbeat-run-summary.test.ts src/__tests__/agent-conversations.test.ts`: 12 files, 627 tests passed. - The new cases check: - Held turns: the run is a candidate, the check posts the exact final answer once, the presentation decision is recorded, the run leaves the candidate set, and a second recovery returns false with one comment. - Live already presented: no candidate, no second comment. - Own question card: no comment, `presentationDecision: none`, the card stays pending, the conversation settles to `waiting`, and the run leaves the candidate set. - Previous-release live decision `none` (`yielded_control_plane_wait`): kept, not re-presented. - `tsc --noEmit -p server`: no errors in changed files. Four unrelated errors remain in this local worktree (`esbuild` not linked in `native-harness-prestage.test.ts`, and a linked-package type mismatch in `run-dispatch/adapters/postgres.ts`). - Turn start: no change. The helper runs only in run completion (after the provider turn ends) and in background presentation recovery. Nothing new runs before `turn.start`, and no files move. The live path keeps the same queries: it short-circuits on `conversation_turn_finished`, and otherwise runs the same card read as before. ## Risks - Low to medium. Recovery can now post the answer of a conversation turn whose task is held. This matches the live path since #20179. - A card-presented turn found by recovery is now settled with `presentationDecision: none`. Before, it stayed a pending candidate and kept the presentation worker (and idle-sleep checks) busy. - A `conversation_turn_finished` turn keeps the `in_progress` check. Held turns accept any status their gate committed. The unsettled-turn checks (active conversation state, session generation, execution run) still block stale or superseded turns. - Previous-release state: no persisted shape changes. A run committed by the previous release with no saved presentation decision (the restart window) is recovered by the new code. The test fixture uses the unchanged finalizer, so it saves exactly the previous release's shape. A run whose live path already recorded a decision keeps it. A held turn that the previous release recorded as `yielded_control_plane_wait` is not re-presented, so no late duplicate or stale reply appears. - Not changed: generic stranded-work recovery in `recovery/service.ts` still skips only `conversation_turn_finished` runs during the publication window. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 01:08:10 -06:00
Nicky LeachandClaude Opus 5.5	990de1e0bf	perf(environments): release a stopped plugin task once (#20203) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugin environment drivers with `supportsTasks` run agents as provider tasks, and warm runs keep a parked task for the next run > - A follow-up run can find that its parked task already stopped. Core then retires that task before it acquires a new one > - Core sent `stop`, `status`, and `complete` in the settle step. Then the plugin release hook sent `stop`, `status`, and `complete` again for the same task. Every call was on the critical path of the new run > - This pull request removes the repeated calls and keeps every call that does work no other call does > - The benefit is a faster start for a run whose parked task already stopped ## Linked Issues or Issue Description No issue exists. This follows the enhancement template. **What existing behavior does this improve?** The release of a plugin task lease that has already stopped. This occurs when a follow-up run finds that its parked warm task stopped. **Subsystem affected** Server: plugin environment drivers and the plugin Runner task lifecycle. **Current behavior** Core sends `stop`, `status`, and `complete` in the settle step. Then the plugin release hook sends `stop`, `status`, and `complete` again for the same task. Core also sends `stop` after resume already saw that the task stopped. On a hosted test deployment, 8 of 40 follow-up runs took this path. From the resume to the new acquire, these runs spent P50 1014 ms and P90 1080 ms. These 8 runs are the full overall P90 (1275 ms). **Proposed behavior** Core does not call the release hook when settle confirmed the stop and completed the same task. After a resume with no lease, Core sends `status` first, and sends `stop` only when `status` does not confirm the stop. **Reason and benefit** The repeated calls do no new work. Removing them saves about 355 ms, plus the `stop` share of about 187 ms, for each such run. These numbers are modeled from trace timings. **Breaking changes** None for the plugin API. A task driver must release task-scoped resources in its `status` or `complete` handling. The doc now says this. **Additional context** The settle `status` stays. Providers release task storage there, and it records the stop that gates the credential writeback and `complete`. ## What Changed - `settleDispatchedEnvironmentTask` now returns `"completed"` when it confirmed the stop and completed the task in this call, `"stopped"` when the lease was already settled, and `false` otherwise. - The plugin driver release does not call `environmentReleaseLease` when settle returned `"completed"`. In all other cases it calls the hook as before. - When `environmentResumeLease` returns no provider lease, the release sends `status` first. It sends `stop` (and a second `status`) only when that `status` does not confirm `executionStopped: true`. Startup and startup-cleanup leases keep the old `stop` first order. - The cleanup sweep (`retryPendingSandboxTeardown`) and `destroyRunLease` do not change. A settle failure still leaves the lease in `pending_cleanup`, and the sweep runs `stop`, `status`, `complete`, and the release hook. - `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` tells drivers about the new order and that task-scoped resources must be released in `status` or `complete`. ### Dispatch count for a stopped parked task (resume to new acquire) | | Calls | | --- | --- | | Before | resume, stop, status, (credentials), complete, release hook (stop, status, complete), acquire: 9 provider calls (10 with Codex writeback) | | After | resume, status, (credentials), complete, acquire: 4 provider calls (5 with Codex writeback) | The modeled saving is about 540 ms for each such run: about 355 ms for the repeated release, and about 187 ms for the settle `stop` and `status`. This pull request keeps the settle `status`, so the real saving is about 355 ms plus the `stop` share of the 187 ms. These numbers are modeled from trace timings. They are not measured on this branch. ### What covers each removed call - Release hook `stop`, `status`, `complete`: settle already sent `stop` and `status` and checked `executionStopped: true` for the same task id (`lease.providerLeaseId`), then sent `complete`. The skip uses settle's success as its condition, not the code path. - Settle `stop` after a resume with no lease: the next `status` must report `executionStopped: true`. If it does not, Core sends `stop` and `status` as before. ### What each kept call protects - Settle `status`: providers release task storage and cancel preparations here. It also writes `runnerTaskExecutionStopped`, which the credential writeback and the `complete` bookkeeping need. - Credential writeback: Codex can rotate its token. The writeback must land before the new task installs credentials. It still runs inside `complete`, before the new acquire. - Settle `complete`: it retires every run lease that owned the task and clears the dispatch marker. - Release hook when settle did not complete a dispatched task in this call: the provider may still hold the lease. ## Verification - `pnpm --filter @paperclipai/shared --filter @paperclipai/plugin-sdk build` - `pnpm --filter @paperclipai/server typecheck` - `npx vitest run src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts src/__tests__/environment-task-runtime.test.ts` - Related suites: `environment-runtime`, `plugin-runner-task.integration`, `task-creation-recovery.integration`, `heartbeat-plugin-environment`, `native-startup-cleanup-recovery`, `plugin-environment-driver-ready-recovery`, `environment-task-authority.integration`, `native-workspace-warm-reuse`, `remote-workspaces.integration`, `legacy-workspace-restore-recovery`, `plugin-task-source-recovery.integration`, `managed-plugin-environments`. Locally, all of these pass except `heartbeat-plugin-environment` "inherits the instance default environment across companies…". That test also fails locally with the `origin/master` service files, so this change does not cause it. The existing `plugin-runner-task.integration` same-run replacement test now expects no release hook after a completed settle. New tests: - The stopped parked path sends exactly resume, `status`, `complete`, acquire. The `status` goes to the old task. - A resume with no lease for a task that still runs sends `status`, `stop`, `status`, `complete`. - A failure at `status`, `complete`, or `stop` leaves the lease in `pending_cleanup`, and the sweep then sends `stop`, `status`, `complete`, and the release hook. - With an observed stop, the Codex writeback runs before `complete`. A writeback failure leaves the lease for cleanup, and the retry writes back and completes. - A startup-cleanup lease still sends `stop` first. ## Risks - A third-party task driver that releases resources only in `onEnvironmentReleaseLease` would not get that call after a completed settle. The doc now states the requirement. The cleanup sweep still calls the hook after any failure. - A driver that returns no lease from resume while the task still runs costs one extra `status`. The behavior stays correct. - No migration. No API change for plugins. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 23:00:49 -08:00
Nicky LeachandClaude Opus 5.5	9046884811	perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner executes native agent runs. A host can park an idle Runner (release it and freeze it) between runs, and resume it for the next run on the same task. > - A follow-up run that reuses a parked Runner must claim it first. The claim is a bootstrap ticket, a checkpoint flush, and the plugin resume that thaws the Runner. > - Core made the claim only inside `attachRun`, after all of its per-run preparation. The claim does not depend on that preparation, so the two ran in series. > - This pull request starts the claim while Core still prepares the run. `attachRun` then waits for the claim that is already in progress. > - The benefit is a shorter start for every follow-up run on a parked Runner. The model below gives about 285 ms per run at P50. ## Linked Issues or Issue Description No public issue exists. The issue is described inline. **What happened?** A follow-up run that reuses a parked Runner claims the Runner only in `attachRun`. Core calls `attachRun` after backend setup: runner environment, assigned MCP tools, tool authority and the plugin task owner. The claim waits for all of this, but it uses none of it. **Expected behavior** The claim runs at the same time as backend setup. Everything that needs the new run's tool authority stays at attach. **Steps to reproduce** 1. Run a Codex or Claude task on a host that parks Runners between runs. 2. Let the run finish and the Runner park. 3. Send a follow-up. Before this change, the plugin resume starts only after backend setup ends. **Paperclip version or commit** `master` at 7086019d4b. **Deployment mode** Hosted, with parked Runner tasks. ## What Changed - **Transport** (`runnerd-codex-transport.ts`): - New `beginRunAttachment({ runId })`. It runs the attach admission checks, then starts the existing `#claimReleasedRunner` as the held attachment operation. It returns `false` and does nothing when the Runner is not released and confirmed, a claim already exists, or admission fails. - The admission checks move into `#assertRunAttachmentAdmission`. `beginRunAttachment` runs them first, and `attachRun` runs them again. - The next `attachRun` for the same run waits for the early claim. An `attachRun` for a different run, `release` and `turn/start` still fail with `native_runner_run_attachment_in_progress` while the claim is held. - After an early claim, `attachRun` fails with `native_runner_resume_reconciliation_expired` before it registers if the claim deadline has passed. - **Pass-throughs:** `HarnessNativeSession`, `CodexHarnessSession` and the session, harness and transport contracts get an optional `beginRunAttachment`. Each layer does the same identity, protocol-integrity and busy checks that its attach does. - **Core** (`native-session-executor.ts`): - The executor records when the in-memory warm entry it reuses was parked (`released`). - `createRunnerdBackend` gets `beginRetainedRunAttachment`. Core gives it only for that parked in-memory warm entry. - The plugin task owner preparation moves into `preparePluginTaskOwner`. On the early path, Core prepares and rebinds the retained owner at the same time as the runner environment and the assigned MCP tools, then starts the claim. - Cold, adopted, remote warm recovery and restart paths keep the old order. ### What stays at attach, and why - Registration, the quiescence probe, `run.attach` and the authority rotation stay in `attachRun`. Checkpoint registration needs the new run's tool authority epoch, and Core creates that epoch during backend setup. - The admission checks run again at attach, so a state change during preparation still blocks the attachment. ## Verification - `vitest run src/live/runnerd-codex-transport-release-resume.test.ts` (runner): 21 passed. Five new modes: - `early claim`: `attachRun` waits for the held claim and does not route the new run before the claim ends; - `early claim abandoned`: a close after the claim ends fails the waiting `attachRun` with `closed`, and the new run is never routed; - `early claim cancellation`: a close during the claim cancels it, and the next `attachRun` fails with `closed`; - `early claim expired`: the deadline passes during the claim, and attach fails before it registers; - `early claim stale ticket`: the deadline passes after the claim, before attach. - `vitest run plugin-runner-release-resume.integration.test.ts plugin-runner-eviction.integration.test.ts` (server, with real Runner processes from `cargo build --workspace --bins`): 2 files, 51 passed, none skipped. New modes: - `cached held claim` (Codex and Claude): the plugin resume is held, and the test proves Core's tool preparation runs while the claim is in progress; - `cached cancellation` (Codex): a cancellation after an early claim retires the Runner and the run stops cleanly. - `pnpm run typecheck:typescript` (runner) and `pnpm typecheck` (server): pass. - Evidence (modeled, not measured live): traces of 8 resumed runs on parked Runners in a hosted test deployment, on current `master`. The claim takes about 300 ms, and it fits inside backend setup in 7 of 8 runs. The modeled saving is about 285 ms per follow-up on a parked Runner at P50 (range 208–331 ms). I did not run this change on a hosted deployment. - Not run: the full server suite (CI runs it), smoke tests, and the hosted park, claim and attach loop with this change. ## Risks - **New failure mode:** Core now claims the parked Runner before backend setup ends. If that run then fails preparation, or is cancelled before attach, the existing retire path closes the claimed session. The Runner is retired, so that agent's next follow-up starts cold. Before this change, a preparation failure happened before the claim, while the Runner was still parked. - **Ordering:** the retained plugin task owner is rebound to the new run earlier, before Core creates tool authority. Nothing in the owner preparation reads that authority. A failure in tool authority setup now happens after the rebind, and the retire path handles it. - **Scope:** only an in-memory warm entry that was parked takes the early path. Cold launches, adopted Runners, remote warm recovery and restart recovery are unchanged. - **Deadline:** the claim deadline now starts earlier. A very slow backend setup can make the claim expire before attach. Attach then fails with `native_runner_resume_reconciliation_expired` before it registers, the same failure as a slow claim today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, git, GitHub CLI) and subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-08 22:56:08 -08:00