mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
a3e7e63330060f8d177e99bfcb26dddf6d6e8b52
7192
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dotta
|
a3e7e63330 |
Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. **The current fix has a causal red/green check.** On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: **5 passed, 0 failed;170 filtered rows excluded**. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed **25/25**, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, **15 selected accepted-result/restart controls passed**. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete **17-case maintenance-server file**. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed **29/29**. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
365c771030 |
fix(chat): give project and workspace tool results their page links (#20324)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask a chat agent to take them to a project or to the files of a task > - The agent finds the right project or workspace, but it cannot link the page > - The `list_projects` and `list_workspaces` tool results have no page route, and the chat link guidance names no project or workspace format > - The agent then writes a bare path or a shortened directory that the person cannot open > - This pull request adds the page `href` to these tool results and adds the project link format to the chat guidance > - The benefit is that chat replies give a clickable link to the real project or workspace page ## Linked Issues or Issue Description **What happened** In chat, a person asked "Take me to the Launch project and show its open tasks." The agent found the project. Its reasoning said "There's no project page route documented explicitly". The reply wrote the project as a bare path `/COM/projects/<id>`, which is not clickable. In a second chat, a person asked where the files of a task are. The agent named the correct workspace but gave a shortened directory and no link. **Expected behavior** The reply links the project page (`/<prefix>/projects/<id>`) and the task's workspace page (`/<prefix>/execution-workspaces/<id>`) as Markdown links, so the person can open them. **Steps to reproduce** 1. Create a company with a project and a task that has a saved execution workspace. Enable isolated workspaces. 2. In chat, ask the agent to take you to the project, or to say where the task's files are. 3. Look at the reply. It has no project or workspace link. Related: Refs #17985 (an earlier open PR that adds project and goal link wording for newly created projects only). This PR covers project discovery and workspaces and adds tool-result hrefs. ## What Changed - `list_projects` rows now carry `href: /<prefix>/projects/<id>`. Both the native runner tool authority and the MCP project-tools route pass the company prefix. - `get_workspace` and `list_workspaces` now carry `href: /<prefix>/execution-workspaces/<id>`, only while isolated workspaces are enabled. The workspace route redirects to the dashboard when the feature is off, so no href is offered then. - The chat link guidance (`AGENT_CHAT_DIRECTIVE`) adds the project link format. It tells the agent to link a returned `href`, and to name a workspace that has no `href` instead of linking it. - Regression tests: guidance text against the registered UI route, `callProjectTool` href, and an embedded-Postgres authority test for project hrefs and the workspace flag on/off. ## Verification - Red then green: the three new tests fail on `origin/master` source and pass with this change. - `npx vitest run` in `server/` for: `agent-conversations`, `task-workspace-api`, `project-tools-pagination`, `task-workspace-tools`, `chat-project-tools`, `dot-runner`, `heartbeat-list` (prompt-size limits), `task-markdown`, `muse-native-visibility`, `paperclip-runner-tool-authority`, `runner-api`, `codex-local-execute`, `board-access-denial`. All pass. - `npx tsc --noEmit -p server` passes. - Live chat eval: see the comment on this PR for the paid runner E2E run of `navigation-preferences.find-project-view` and `workspaces.locate-files` with this change. ## Risks - Low risk. The change adds one optional field to read-only tool results. The output schemas allow extra fields. - The guidance grows by two sentences. The prompt-size test in `heartbeat-list.test.ts` passes. - `list_projects` and the workspace tools now do one extra small read (company prefix, and the instance experimental settings for workspaces). ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
0f84df91da |
fix(native): accept a completed child's file in the parent's completion report (#20325)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In onboarding, the agent plans the user's first task, creates one child task for the work, and waits for it > - The `first-task` skill tells the agent to publish a requested file on the child task. When the parent wakes, the agent links the child's file and does not publish a copy > - Native `register_deliverable` is bound to its current task, so the parent cannot register the child's file again > - The native completion check required each cited attachment to be on the task being completed > - So when the parent cited the child's download link, `paperclip_finish` failed, and the agent reported the finished onboarding task as blocked > - This pull request lets a completion report cite an attachment on a direct child task that is done > - The benefit is that onboarding closes after its child task delivers a file, and the user does not see a false "blocked" status ## Linked Issues or Issue Description Related: #18258 (onboarding artifact completion instructions). **What happened?** An onboarding run created a child task to build and publish a project. The child task registered `project.zip` with `register_deliverable` and was marked done. The parent woke with reason `issue_children_completed`. The agent called `paperclip_finish` with `reportedWorkDisposition: "done"` and cited the child's `/api/attachments/<id>/content` link. The call failed two times with: ``` Completion cites no registered attachment on this task. Use register_deliverable for the requested file and cite deliverable:<attachmentId> from its receipt. ``` The skill does not let the agent register the file on the parent, so the agent called `paperclip_block` with "Paperclip completion validation" as the owner. The parent task stayed `blocked`. Nothing wakes it again. **Expected behavior** The parent completes and links the child's file. **Steps to reproduce** 1. Create a task with a child task. 2. In a native run on the child, register a file with `register_deliverable`, and mark the child done. 3. In a native run on the parent, call `paperclip_finish` with `reportedWorkDisposition: "done"` and the child's `deliverable:<attachmentId>` or `/api/attachments/<attachmentId>/content` as evidence. 4. The call fails with "Completion cites no registered attachment on this task". ## What Changed - `server/src/services/native-runtime/native-deliverable-feedback.ts`: when a cited attachment is not on the task, `validateNativeDeliverableEvidence` also looks for it on a direct child task with status `done`. If it is there, the citation is accepted. - The child's file is context, the same as an attachment from an earlier run on the same task. It does not count as this run's publication of a file that the parent's own request names. - With `lockSources`, the finalization path locks the attachment and child task rows with `FOR SHARE NOWAIT`, as it does for other sources. - The error text now says "on this task or its completed child tasks". - This applies to the `paperclip_finish` check and to the two finalization checks, because they use the same function. - `doc/AGENT-ARTIFACTS.md`: the native runner section states that a completion report can cite a file on a completed direct child task. ## Verification - Extended "prepares one verified same-run attachment and replays without duplicates" in `server/src/services/native-runtime/native-runner-file-handoff.test.ts`: - An attachment on an unrelated task is still rejected. - An attachment on a child task that is not done is still rejected. - An attachment on a done child task is accepted, through `paperclip_finish` and through the locked finalization path. - When the parent's own request names a file, the child's file alone does not satisfy it. - Without the source change, the test fails with the same error as the run above. - `npx vitest run` on `chat-attachment-reuse`, `native-completion-feedback`, `native-completion-format-feedback`, `native-deliverable-feedback`, and `native-runner-file-handoff` tests in `server/src/services/native-runtime`: 281 passed. - `npx vitest run src/__tests__/native-result-commit-work.test.ts src/__tests__/native-status-arbiter-corpus.test.ts` in `server`: 167 passed. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Only direct children with status `done` are accepted. A grandchild's file, or a file on a child that is not done, is still rejected. - A parent that names a requested file in its own request still needs its own delivery evidence. This keeps the current rule for earlier outputs. - A task that is already blocked by this error does not change. The user can mark it done, or comment so the agent runs again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (code reading, database queries, shell commands, and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
47dccbf3eb |
fix(recovery): re-prove an exhausted save's agent stop and free the workspace (#20295)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run ends, Paperclip saves the files that the run changed in its remote workspace. Until that save settles, the run holds the workspace, so no other task can write it > - When the save fails, a bounded system recovery proves that the agent stopped (a provider stop receipt) and then saves the files again > - If the stop cannot be proven before the 30-minute recovery window ends, the recovery is exhausted with `agent_stop_unconfirmed`. Exhaustion was terminal: Paperclip never proved the stop again, even after the run ended and its sandbox had a stop receipt > - The hold then stayed forever. On staging, a child task's save was exhausted this way on Nov 5. Its parent and its two sibling tasks share that workspace, and all of them stayed blocked for four days > - This pull request re-arms such an exhausted save automatically, on a bounded backoff, once the run is provably over. After the bounded re-arms, the hold continues from the last save with the unsaved files attached > - The benefit is that a dirty workspace hold always settles without a person, and no unsaved work is discarded silently ## Linked Issues or Issue Description Refs #20273 (the stall sweep stops failing wakes against this kind of hold) Refs #20278 (a settled restore hold returns its warm sandbox) Refs #19860 (subtasks get their own workspace) **What happened?** A child task's run wrote its remote workspace and failed to save it (`native_workspace_sync_out_retry_exhausted`). The save recovery made one attempt. That attempt failed with `agent_stop_unconfirmed`, and the recovery window ended before a second attempt. The recovery became `exhausted`. Ten minutes after the first attempt, the sandbox lease got a provider stop receipt, so the agent was provably gone. But nothing looked at the exhausted recovery again. The run kept the workspace's writer hold. The parent task and every other task on that workspace could not start ("The workspace still has an unfinished writer or save operation."). The blocker told a person to press Retry. **Expected behavior** An unproven stop is a wait, not a refusal of the saved files. When the run is over, recovery must prove the stop again and finish the save, so the hold settles automatically. If the stop still cannot be proven, Paperclip must keep the unsaved files and then release the hold. A person must not need to act. **Steps to reproduce** 1. Run a task in a remote (sandbox) workspace. Make its final save fail. 2. Keep the workspace recovery from proving the agent stop until its 30-minute window ends (for example, keep the finalization ownership busy). 3. See that the recovery action is `exhausted` with `agent_stop_unconfirmed`, and that it stays so after the lease gets a stop receipt. 4. Wake another task bound to the same workspace. See that its setup fails because the workspace is held. **Root cause** - `server/src/services/workspace-recovery.ts` (`recoverRetainedWorkspace`): an `exhausted` state only goes to the sibling settlement and to the retained-source fallback. The fallback accepts only permanent source refusals (`isUnverifiableSourceFailure`). Every other exhausted state is handed to the board and never attempted again. - `recheckStoppedWorkspaceRecovery` skips `exhausted` states, and it only accepts acknowledged cancellations. A failed run whose lease got a stop receipt later is never re-examined. - `retainedWorkspaceCandidates(…, obligationsOnly)` and `nextWorkspaceRecoveryAt` do not select exhausted `agent_stop_unconfirmed` holds, so the recovery worker never wakes for them. ## What Changed - New `server/src/services/workspace-recovery-stop-proof.ts` (`rearmExhaustedStopProof`). For an exhausted save whose last failure is `agent_stop_unconfirmed` (`recheck_evidence`), it starts a new bounded system recovery generation. It does this only when the source run is provably over: the run is terminal, no local execution remains, and no controller lease or finalizer lease is live. The new generation repeats the strict stop proof (a provider stop receipt) and the save. It records an `issue.workspace_recovery_rearmed` system activity. - Re-arms are bounded: at most 4. The first is due at once. The next ones are due 30 minutes, 2 hours and 8 hours after the previous re-arm. The `stopProofRecovery` marker in the recovery action evidence records the count and the next due time. - `workspace-recovery-fallback-discovery.ts`: new discovery conditions (`stopProofRearmPendingCondition`, `stopProofRearmDueAt`, `stopProofRearmDueCondition`) and `retainedHoldFallbackEligible`. `retainedWorkspaceCandidates` (worker pass) and `nextWorkspaceRecoveryAt` use them, so the recovery worker wakes when a re-arm is due. - `workspace-recovery-fallback.ts`: when every re-arm is spent and the stop is still unproven, the hold takes the existing retained-source fallback. The unsaved files are attached to the task (unapplied, secrets omitted), the unverified epoch is abandoned, and the task continues from the last save. - `system-recovery-state.ts`: the exhausted `agent_stop_unconfirmed` text now says that Paperclip tries again automatically. It no longer asks a person to press Retry. Retry stays available. - Review follow-ups: the re-arm applies to every workspace recovery the worker owns (restore and historical holds too). A due re-arm is scheduled no earlier than the source run's controller or finalizer lease expiry, so the worker is not woken every 15 seconds while a lease is live. A fallback whose bounded checks are spent is reconsidered once when an allocation of the source run changes, and `settleStopOnlyCleanup` wakes the workspace recovery worker when it records a stop receipt. The automatic-retry text is shown only while a re-arm remains. - `doc/execution-semantics.md`: documents the re-arm, its bounds, the fallback, and the upgrade behavior. ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - **Carried forward, no migration.** A hold exhausted by an older release has no `stopProofRecovery` marker. The recovery worker treats it as due at once and re-arms it on its first pass after deploy. The new regression test saves the exact Nov 5 evidence shape (`attempts: 1`, `retriesUsed: 0`, `state: exhausted`, no marker) and resumes it with the new code. - **Mid-run deploy.** A run that is still running, or whose controller lease or finalizer lease is live, is not re-armed. The test covers a live controller lease. The re-arm uses the same CAS (generation and revision) and the same row locks as the other recovery writers, so an old worker's late write cannot overwrite the new generation. - **No new refusal codes.** No saved run identity changes. The re-arm writes only recovery-action evidence. ## Verification - Red (before the fix), `npx vitest run src/services/workspace-recovery-stop-proof.test.ts`: 3 failed, 1 passed. The staging-shape test failed because the worker did not wake (`nextWorkspaceRecoveryAt` returned `null`) and the hold stayed. The backoff and fallback tests failed for the same discovery gap. - Green (after the fix): 4 of 4 pass. - Staging shape: a parent and 3 children that share one remote workspace. One child's save is exhausted with `agent_stop_unconfirmed`, and its run ended days ago with a stop receipt. Before recovery, the hold refuses the parent (`remoteWorkspaceHoldRefusesStart` is `true`). The recovery worker discovers the hold, re-arms it, proves the stop, saves the files and completes. The workspace is not dirty, it has a new snapshot, and the hold is `free` for the parent. The action stays system-owned and its text asks no person to act. - A live controller lease keeps the hold. No re-arm happens and no lease is resumed. - A re-arm that is not yet due is not discovered. The worker's next wake is in the future, and the text says that Paperclip tries again automatically. - After 4 re-arms, the hold continues from the last save. The unsaved files are captured as a task attachment, and the hold is `free`. - A live controller lease defers the worker's next wake to the lease expiry, and the hold is not selected before it. - A spent fallback is not selected, and a later lease change (a late stop receipt) makes it selected again; the worker then settles it. - `system-recovery-state.test.ts`: the automatic-retry text appears only while a re-arm remains. - Related suites, all pass: `src/services/workspace-recovery*`, `src/services/remote-workspace*`, `src/services/system-recovery-state.test.ts`, `src/services/task-retry.test.ts`, `src/__tests__/heartbeat-observed-workspace-preservation.test.ts` (34 files, 1510 tests), plus the stop-cleanup suites (`environment-retained-orphan-sandbox`, `live-recovery-work`, `retained-native-session-cleanup-work`, `legacy-workspace-restore-recovery`, native workspace warm-reuse, cancelled preservation, reconstruction and failed recovery; 11 files, 622 tests). - `pnpm verify`: exit 2 ("broader verification required", because `doc/execution-semantics.md` is a global CI input). The focused suites above cover the change, and required CI supplies the broad coverage. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A re-armed generation resumes the stopped sandbox to save its files. This is the same physical path as a manual Retry. It runs only when the source run is provably over. - Re-arms are bounded (4) and spaced, so a stop that can never be proven does not loop. After that, the existing retained-source fallback settles the hold. If the retained files cannot be read, the fallback records why. This is the same behavior as for a refused source. - The parent's workspace receives the child's saved files, because the child (bound by an older release) wrote that workspace. This is the existing #19860 behavior: the files that a legacy child left in the shared workspace stay there. #20314 moves the stranded legacy children to their own workspace. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only API checks) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8fe524f796 |
fix(issues): let the board confirm an unidentified run stopped instead of an endless Retry wait (#20310)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a new attempt can start on a task, Paperclip must prove the previous run's agent stopped, so two agents never work the same task at once > - A local native run can be interrupted after it asked to start the runner but before it recorded the runner's process. The runner may have saved state and started a provider, and provider processes can outlive it, so the existing check (#19694) rightly refuses to assume the run stopped > - Today that case has no way out: every Retry is accepted and then waits forever as `process_identity_missing` (storage chaos ST1, t02/t03) > - This pull request makes the wait explicit and actionable. The task explains why it waits, and a board operator who has checked can confirm the run stopped; the parked Retry then starts at the click > - A second shape from the Daytona storage baseline (t06) parked a Retry with no reason at all: a follow-up was interrupted by a graceful shutdown while it was still restoring its workspace. No provider ran, but the restore hold kept the same agent held, so the Retry never started > - The benefit is that no Retry stays accepted-and-waiting with nothing anyone can do ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (ST1), tasks t02 and t03. **What happened (CHA-3, run `3c3f6241`):** - The run requested a runner launch (`native.process_start_requested`), and a K2 restart hit before its process identity was recorded. - After the restart, the resumed attempt failed on the storage outage. The task was blocked. - The board's Retry (202) was parked as `executionWait.reason = process_identity_missing`: "The previous run has no verified stop record." - `recordUnidentifiedLocalRunnerStop` cannot prove the stop because this run's runner saved control-plane state. Once that exists, a provider may have started (#19694), and a provider is not guaranteed to carry `--run-id` or `PAPERCLIP_RUN_ID` (the Codex provider env omits it on purpose). The shared owner prover needs a PID, which the run never recorded. - Nothing could ever clear the wait. **Daytona storage baseline `w-20261109T154810-s9911`, task t06:** - The run succeeded. The follow-up was cancelled (`continuation_task_ownership_changed`), and the next run was interrupted by a graceful shutdown (`server_shutdown_interrupted`) 3 s into `preparing`. It was restoring the saved remote workspace and had launched no provider. - Restore recovery saved the files (`workspaceRecovery.status: recovered`, system recovery `complete`). But the `legacy_execution_requires_reconciliation` restore hold releases only for a different owner, or for a native Stop fenced before any provider turn, so it kept the same agent held. - The UI Retry returned 202 and parked with `executionWait.reason: execution_recovery`, the generic wait, so the task showed a Retry that never started. Automatic proof needs the runner to record its identity before saving state and to group provider processes under it. That is follow-up runner work; this PR removes the endless wait now. ## What Changed - `server/src/services/native-local-process-stop.ts`: new `confirmUnidentifiedLocalRunnerStopped`. It accepts only a finished native run of the task with no recorded process and whose latest launch evidence is an unconfirmed launch on this host in a local environment. It still refuses while a runner with `--run-id <run>` is running. It records `native.local_process_stopped` with `basis: "operator_attested"` and the attesting user. That is the receipt the existing continuation gate already accepts. - `server/src/routes/issues.ts`: `POST /issues/:id/recovery-actions/confirm-run-stopped` (`runId`, `stopEvidence` of 20 to 12,000 characters). It is board-only and requires issue read plus `runtime:confirm_owner_stop`, like the other stop attestations. It logs `issue.run_stop_confirmed` with the evidence, promotes deferred intents for the issue, and resumes each parked task Retry, so work starts at the click. Refusals return 409 with a code. - `packages/shared`: `confirmRunStoppedSchema`, and the presentation user action `confirm_run_stopped`. - `server/src/services/task-interruption-presentation.ts`: a Retry parked as `process_identity_missing` for a native run that recorded no process now presents as waiting for the user (`canRetry: false`, `userAction: "confirm_run_stopped"`), with a message explaining why and who can act. A run with a recorded process keeps the existing verification wait. - UI: - `ExecutionBlockerNotice` shows the explanation and, for users who can confirm a stop, "Confirm it stopped" with a short evidence form. - It reuses `PreviousServerStopConfirmation`, now with optional `action`, explicit `runId` and `formId`, and a custom button label. - The Inbox names the action. - `server/src/services/execution-blocker.ts`: `interruptedPreparationBeforeProviderWorkCondition`. The restore hold also releases for the same agent when the stopped run was a system-interrupted preparation (legacy runtime never resolved, still `preparing`) with no process identity, no runner or provider events, no finalization or result, and every lease released with a confirmed stop. It mirrors the existing interrupted-preparation proof and still requires the completed workspace-recovery receipt. Auto-replay is unchanged; this only lets the user's Retry or message start. - Review fixes: - `unidentifiedLocalLaunchRefusal` holds the endpoint's launch, host and lease checks, and the presentation offers `confirm_run_stopped` only when it passes. - The attested stop wakes cost accounting in the same transaction. - The confirmation form is keyed by task and run. - The route is registered in OpenAPI. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-confirm-run-stopped-routes.test.ts`: - The parked Retry presents `confirm_run_stopped` with the explanation. - Confirming records the operator-attested stop and the activity entry, satisfies the stop check, and resumes the parked Retry: its receipt is coalesced into a newly started run. - It refuses with 409 while a runner for the run is running. - Agents and viewers get 403. - `ui/src/components/ExecutionBlockerNotice.confirm-run-stopped.test.tsx`: the notice offers "Confirm it stopped" (and no dead Retry) and posts the run and evidence. A viewer sees the explanation without the action. - `server/src/__tests__/interrupted-preparation-restore-retry.test.ts` (t06 shape): - The restore hold no longer blocks. - A Retry is accepted and resumed into a started run, and the audit hold keeps `replay: blocked`. - The hold stays when a provider turn started, when a runner event was recorded, when an operator stopped the run, or when the sandbox has no stop receipt. - `issue-confirm-run-stopped-routes.test.ts`: - A case per refusal code checks the presentation keeps the generic wait and the endpoint returns 409. - An `alreadyStopped` case. - An accounting-wake assertion. - The UI test switches targets with stale evidence. Regression and static checks: - `ExecutionBlockerNotice`, `WorkspaceExportRecovery` and its control tests: 74 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A board operator can now attest that an unidentified run stopped. The attestation is permission-gated, requires written evidence, is activity-logged, and is refused while a runner for the run is visibly running. A wrong attestation could start a second agent beside an orphaned provider, the same trust model as the existing previous-server attestations. - The confirmation component now accepts a run without a recovery action. Existing callers pass the action as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ff2d463eed |
feat(cli): manage agent grants and keep remote login on the public origin (#20323)
## Thinking Path
> - Paperclip lets people control their AI agents and permissions.
> - Remote operators need a browser-approved CLI login.
> - Proxies can send that login's approval link to an internal host.
> - Human member commands cannot grant an agent an explicit permission
such as commentary:view.
> - This change keeps approval on the canonical public origin and adds
Board-only single-grant commands.
> - The API checks company access and permission-management authority
and records each change.
> - An operator can sign in, grant one agent permission, and preserve
the agent's other grants.
## Linked Issues or Issue Description
**What happened?**
A proxied CLI login returned an approval URL on the upstream host. The
signed-in browser was on the public host. The CLI also had no command to
add an arbitrary explicit agent grant: member commands manage human
members, while agent permissions:update exposes fixed configuration
fields.
**Expected behavior**
Approval uses the configured canonical public origin. A Board operator
with users:manage_permissions can grant or revoke one explicit agent
permission without replacing other grants or activating an inactive
membership.
**Steps to reproduce**
Configure a public auth origin. Create a CLI challenge through a proxy
that replaces forwarded-host headers. Inspect the approval URL. Then try
to grant commentary:view to an agent through the CLI.
## What Changed
- Use the existing public-origin resolver for CLI approval URLs.
- Add agent permission:grant and permission:revoke commands.
- Add PATCH /api/agents/{id}/permission-grants with strict shared
validation and Board-only OpenAPI metadata.
- Serialize each mutation on the agent row. Preserve unrelated grants
and membership state. Commit the grant and its activity record together
and publish activity after commit.
- Document remote login and single-grant management.
## Verification
- The new approval-link regression failed before the fix and passes
after it.
- All 25 CLI auth route tests pass.
- All 72 affected CLI, login and OpenAPI tests pass.
- All 5 PostgreSQL grant tests pass, including actual commentary reads
(403 before grant, 200 after grant, 403 after revoke), preservation,
audit attribution, concurrent additions, invalid inputs, inactive or
missing membership, agent self-grants, cross-company denial, and
rollback on audit failure.
- Shared and CLI typechecks and server declaration-layer checks pass.
- pnpm verify requests broad CI coverage because doc/CLI.md is a global
verification input. Greptile reviewed final head
|
||
|
Dotta
|
1b7fe33432 |
fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. **What happened?** Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. **Expected behavior** Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. **Steps to reproduce** 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. **Paperclip version or commit** The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. **Deployment mode** Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
0cf70c9d0c |
fix(chat): tell overviews to state when the picture is from (#20301)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People ask the CEO chat agent for a quick picture of what the company is doing now > - Such a picture is only useful when the person knows how current it is > - The chat guidance asks for accurate status, but it does not ask for a time > - A live overview said "right now" and gave no time, so the person could not tell when it was observed > - This pull request adds one guidance sentence: state the picture's time as a UTC clock time, from the latest message timestamp > - The benefit is that overviews tell the person when their information is from ## Linked Issues or Issue Description Refs #19614 **What happened** A person asked the chat agent "Give me a quick picture of what the company is doing right now." The agent read agents, projects, tasks and approvals and gave a correct summary. The summary did not say when it was observed. The only time word was "right now". **Expected behavior** An overview of current work states when the picture is from, as a clock time. The native prompt already lists each new comment with its timestamp, so the agent has a time it can use. **Steps to reproduce** 1. Open a chat with the CEO agent in a company that has active, blocked and done tasks. 2. Ask "Give me a quick picture of what the company is doing right now." 3. The reply summarizes the work but gives no observation time. ## What Changed - `AGENT_CHAT_DIRECTIVE` status-update guidance: when the agent describes what the company, a project or agents are doing now, it states a UTC clock time, such as "As of 15:26 UTC". It uses the latest message's timestamp unless a record it read gives a later time. - A regression test checks that the directive contains this guidance. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-list.test.ts` passes (includes the prompt-size limits). - The chat-catalog eval `activity.company-overview` now grades a real observation time (stacked eval PR on `chat-eval/master-candidate`). This fix is cherry-picked onto that eval branch for the next live run. No live run has used it yet. ## Risks - Low risk. This is one guidance sentence. Overviews get one extra clause with a time. - The time is the message time, not a server clock read. Reads happen seconds after the message, so the difference is small. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
12523fb677 |
fix(chat): send a task whose automatic recovery stopped to the person's Retry (#20306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person asks the chat agent (the controller) to act on tasks that other agents own > - When a task's automatic recovery stops, for example after three runner losses, the task is blocked and the person gets a Retry control on it. Retry rechecks the task's gates and keeps the failed attempt's evidence > - The blocker told every reader "Try again or send a new message to continue". A controller read that as its own instruction. It found that Retry is board-only, then resumed the task with a comment and reassigned the task to itself > - That skips the assignee's budget and hold gates and hides the failed-run Retry from the person > - This pull request names Retry as the person's control in the blocker text, and tells the controller to point the person to it > - The benefit is that "try that task again" leads to the supported, gated Retry instead of an agent takeover ## Linked Issues or Issue Description Refs #20270 (automatic recovery after a lost local runner) **What happened?** In the chat eval `chat-features.runner-opencode.local.recovery.retry-exhausted`, a person asked the CEO agent to "Try the website task again" after the website task's native recovery was exhausted. The agent read the task's `executionBlocker` (`canRetry: true`, nextAction "... Try again or send a new message to continue once the previous execution has stopped."). It found that `POST /api/issues/{id}/retry` is board-only. It then posted a `resume: true` comment, which moved the task from blocked to todo, and reassigned the task to itself to re-run it. The task lost its failed-run Retry control, and the original assignee's gates were not checked. **Expected behavior** The agent links the original task, says why it stopped, and tells the person to use Retry there. It takes the work over only when the person explicitly asks for that. **Steps to reproduce** 1. Let a native run exhaust its automatic recovery (for example, three runner losses). 2. In the controller chat, ask it to try that task again. 3. The agent comments to resume the task, or reassigns it, instead of pointing to Retry. ## What Changed - `execution-blocker.ts`: the text that is added when a person can retry is now `EXECUTION_BLOCKER_PERSON_RETRY_ACTION`: "A person can use Retry on this task, or send it a new message, to continue once the previous execution has stopped. Agents cannot retry it for them." - `agent-conversations.ts`: a new chat guidance paragraph. When another task's automatic recovery has stopped, the supported next step is the person's Retry on that task. The controller links the task and explains why it stopped. It does not reassign the task, change its status, comment on it to resume it, or create a replacement task, unless the person explicitly asks it to take the work over. ## Verification - `board-access-denial.test.ts`: the directive contains the new rule. - `execution-blocker-read.test.ts`: a retryable blocker's nextAction ends with the person-Retry text. - 54 tests pass locally for these two files. - Paid live run of `recovery.retry-exhausted` with this change and #20270 on the eval branch: the run ID will be added below. ## Risks - Low. Only text changes. Any UI that shows the blocker's nextAction shows the new sentence instead of "Try again or send a new message ...". - No schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
97188f276c |
perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.
## Linked Issues or Issue Description
Refs #20190, #20285
**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).
**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.
**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.
## What Changed
- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
- New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
- The other `DurableRunnerConfig` literals set `turn_scoped: false`.
## Verification
- `cargo test -p paperclip-runner-core`:
- `--lib`: 602 passed.
- `--test acpx_park_preparation`: 22 passed.
- `acpx_park_readiness`: 8 passed.
- `acpx_warm_attachment`: 4 passed.
- `acpx_provider_suspend`: 5 passed.
- `acpx_provider_session`: 12 passed.
- `acpx_provider_turns`: 29 passed.
- `native_provider_backend`: 47 passed.
- `durable_recovery`: 3 passed.
- `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(
|
||
|
DottaandClaude Opus 5.5
|
189fefb991 |
perf(task-chat): warm the whole first frame before a task opens (#20286)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from the issues list and the inbox to continue a conversation with an agent > - The compact task chat shows its first frame only after about ten reads settle. These include recent comments, activity, recent runs, interactions, attachments and live runs > - Navigation prefetch warmed only the issue and its recent comments. The other reads started on the click, and the slowest one decided when the chat appeared. On staging that took 1.1–2.0 s > - This pull request warms the whole first-frame set when an open is likely, under the exact keys the page reads > - The benefit is that a task the person is about to open can render from cache. A pointer that only passes over rows does not add load ## Linked Issues or Issue Description **Problem or motivation** On a busy instance, the composer and the latest messages appear 1.1–2.0 s after a click on a task row. The page waits for about ten reads. Before the click, only two of them are warmed. **Proposed solution** Warm the rest of the first-frame reads when an open is likely: a mouse or pen press, a click, keyboard Enter, or a hover or focus that rests on the link for 120 ms. The page and the prefetch use the same fetchers and query keys. **Alternatives considered** - Warm everything on every hover. This sends about 7 requests for each row that a pointer crosses, so it was rejected. - Warm only on press. The time from press to click is short, so most of the wait would remain. - A single bootstrap endpoint. This is planned separately, because it changes the API contract. **Roadmap alignment** This is part of the task-loading performance plan in #20276. It follows #20277 and #20284. ## What Changed - New `ui/src/lib/issue-chat-history-queries.ts`: - The recent activity and recent run window fetchers moved here from `IssueDetail.tsx`, together with their query keys. The logic is unchanged. - `prefetchIssueChatHistory` warms activity, the run window, interactions, attachments and live runs. It warms the active run only for running work, as the page does. - Live state uses a 5 s stale time. The other reads use the existing 30 s navigation stale time. - The run-window prefetch uses the same conversation mode as the page. Thus a reconnect in an agent conversation reads only the watched runs by id. - `prefetchIssueCommentInputFacts` warms the input facts of the recent comments after those comments arrive. - `ui/src/hooks/useIssueIdBatches.ts`: new `prefetchIssueIdBatches` fills the same batch keys that the hook reads. The hook and the prefetch share one batching function. - `prefetchIssueDetailForNavigation` accepts `history: true` and then also warms that set, plus the comment input facts. - `IssueLinkQuicklook`: - A hover still warms the light set at once. The quicklook trigger does not start a history prefetch. - A hover that rests on the link for 120 ms warms the full set. Leaving the link before 120 ms cancels it. - Keyboard focus uses the same 120 ms dwell as a hover. Blur cancels it. - A primary mouse or pen press and a click warm the full set at once. - A touch can start a scroll, so it warms only the light set. The full set waits for the click. - The keyboard Enter handlers in `IssuesList`, `LegacyIssuesList`, `Inbox` and `LegacyInbox` pass `history: true`. - `IssueDetail.tsx` uses the shared fetchers and keys. ## Verification - `cd ui && npx vitest run src/lib/issueDetailCache.test.ts src/components/IssueLinkQuicklook.test.tsx src/hooks/useIssueIdBatches.test.tsx` passes 29 tests. These are the new ones: - A `history` prefetch fills every key that the compact chat waits on, with the same run-window shape that the page builds. - A `history` prefetch fills the comment input-fact batch keys that the chat reads. - In an agent conversation, a reconnect reads the watched runs by id, not every run. - Prefetched id batches render with no loading state on the first render. - The active run is read only for work that is running. - A light prefetch makes no history requests. - A pointer that passes over a link in less than 120 ms warms only the light set, with the quicklook on and off. - A hover that rests on a link warms the full set. - A mouse press warms the full set at once. - A touch warms only the light set until the click. - Keyboard focus warms the full set only after it rests on the link. - The tests for the review fixes (quicklook hover, input facts, conversation mode, touch and focus) fail without the fix. - The `IssueDetail`, `Inbox`, `IssuesList` and `LegacyInbox` suites pass: 4 files, 412 tests. - `pnpm typecheck` in `ui` shows no new errors. The only errors are existing ones from an unbuilt `plugin-sdk` package in this worktree. - The staging harness from #20276 will measure the gain after deploy, for a hover, a click, and a revisit. ## Risks - More requests when an open is likely: up to 7 extra reads for each task that is pressed with a mouse or pen, or hovered or focused for 120 ms. These are the same reads that the page would send on open, so a completed open costs nothing extra. A hover that does not lead to an open costs those reads. - Freshness: a prefetched result can stay fresh for up to 30 s (5 s for live state). Websocket invalidation still refreshes these keys, as on a revisit today. - No API or schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
90404d2033 |
perf(run-logs): read object-store log tails in parallel, once, and cached (#20284)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted instances keep run logs in object storage as a journal of
content-addressed segments and binary index nodes
> - The task chat reads the last 32 KB of a run's log, aligned to a
record, to show what the agent did
> - A run that flushes often has many small segments. The tail read
fetched each object one after another, and record alignment walked the
index a second time
> - On staging, one tail read took 11–12 s, and the task chat waited for
it (see #20277)
> - This pull request reads the tail in parallel, in one pass, and
caches the objects that cannot change
> - The benefit is that the first tail read is about 6.6x faster, and
repeated reads (live polling, reopening a task) use only the mutable
head
## Linked Issues or Issue Description
**What happened?**
`GET
/api/heartbeat-runs/:runId/log?offset=…&limitBytes=32000&alignStartToRecord=true`
took 11–12 s on staging for a 400 KB object-store log. Without alignment
it took 6.1 s. A read from offset 0 took 2.0 s. The task chat waited for
this read before it showed the run's output.
**Expected behavior**
A 32 KB tail read returns in well under a second, and a repeated read of
the same tail costs about one storage round trip.
**Steps to reproduce**
1. Use object-backed run logs (`s3_segments`).
2. Write a run log in many small flushes, as a live run does.
3. Read its last 32 KB with `alignStartToRecord=true`, and count the
object GETs and how many run at the same time.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
099adb2728 |
fix(recovery): settle a remote workspace save that stopped without finishing (#20291)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.
**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.
**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
ef2b8d5324 |
fix(issues): make every Retry start work or say why (#20283)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a task's run fails, the board recovers it with a Retry button, so that button must either start work or say why it did not > - An audit of every Retry-type button found three gaps. The task Retry shown during a scheduled retry's countdown returned "accepted" but changed nothing. A stale Retry looked like success in the Inbox. "Retry now" skipped the task's read and change checks, so a member who cannot see a private task could start its retry > - This pull request makes the task Retry start a scheduled retry immediately, makes every Retry that did not start work say why, and gives "Retry now" the same authorization as the task Retry > - The benefit is that no Retry click silently does nothing ## Linked Issues or Issue Description Refs #19831. The board reported a dead Retry button. A read-only audit of every Retry button confirmed: - **Task Retry during a scheduled retry was a no-op.** `TaskInterruptionPresentation.canRetry` is true in `retry_scheduled`, so `ExecutionBlockerNotice` shows Retry. `requestTaskRetry` treats `scheduled_retry` as busy and inserts a `coalesced` wake pointing at that run. `resumeTaskRetry` reads only `deferred_issue_execution`, and dispatch uses coalesced user wakes only to carry comments. Nothing promoted the run: the API answered 202 `accepted: true` and the countdown did not change. - **A stale Retry looked like success.** `accepted: false, stale: true` comes back as HTTP 200. The Inbox ignored `accepted` and opened the task as if work had started. The task page showed no reason. - **"Retry now" checked less than the task Retry.** It ran only `assertBoard` and `getAccessibleResource`, skipping `assertIssueReadAllowed` and `issue:mutate`. A company member who cannot read a private task could promote that task's scheduled retry. - **"Retry now" said nothing for some outcomes.** It toasted only `promoted` and `gate_suppressed`. ## What Changed - `server/src/services/task-retry.ts`: when the Retry coalesces into the task's latest run and that run is `scheduled_retry` (with no other Retry already active), the receipt returns `scheduledRetryRunId`. - `server/src/routes/issues.ts`: - `POST /issues/:id/retry` then promotes that run through the existing `retryScheduledRetryNow`, the same path "Retry now" uses. It logs `issue.scheduled_retry_retry_now` with `source: "task_retry"` and returns `scheduledRetry: { outcome, message }`. - `POST /issues/:id/scheduled-retry/retry-now` now runs `assertIssueReadAllowed` and `decideIssueAccess("issue:mutate")`, matching `/retry`. A private task the member cannot read now answers 404. - `ui/src/hooks/useTaskRetry.ts`: new `taskRetryNotStartedReason`. When a Retry did not start work (stale, or a scheduled retry that is gate-suppressed, waiting, or gone), the hook returns that reason as its error. `ExecutionBlockerNotice` and `StalledDependencyNotice` already display it. - `ui/src/pages/Inbox.tsx`: a task Retry that did not start work shows the reason in the error toast and does not navigate. - `ui/src/hooks/useRetryNowMutation.ts` and `IssueScheduledRetryCard.tsx`: every "Retry now" outcome now shows its headline and message, including `no_scheduled_retry` on the card. - `ui/src/api/issues.ts`: `retryTask`'s response type includes `scheduledRetry`. ## Verification Red/green (each new test fails against master and passes with this PR): - `server/src/__tests__/issue-scheduled-retry-routes.test.ts`: - "starts a scheduled retry now when the task Retry is clicked during its countdown": 202 with `scheduledRetry.outcome: "promoted"`, the same run becomes `queued`, and no second retry run is created. - "refuses Retry now on a private task the member cannot see, like the task Retry": on master "Retry now" answers 200 `promoted`; now both routes answer 404 and the run stays `scheduled_retry`. - The seed now adds the board user's membership, which the stricter route needs. All 33 tests pass. - `ui/src/hooks/useTaskRetry.test.tsx`: a replaced attempt, a gate-held scheduled retry and a waiting scheduled retry each show their reason; a promoted one shows nothing extra. - `ui/src/pages/Inbox.test.tsx`: an Inbox Retry that did not start work (stale, or gate-held) toasts the reason and does not navigate. - `ui/src/components/IssueScheduledRetryCard.test.tsx`: "explains when there was no scheduled retry left to start". Other Retry buttons: the chat failed-run Retry already reports stopped and failed retries (`agents.test.ts`). Agent lifecycle "Retry now" and cost "Retry accounting" start no task work and already surface their API errors. Regression and static checks: - UI: 7 files, 168 passed. Server: task-retry and scheduled-retry suites, 149 passed. `issue-agent-mutation-ownership-routes.test.ts` passed. - `maintenance-server-routes.test.ts` "assembled maintenance server production wiring" fails identically on master on this Mac (`ensure-build-deps`). - `tsc --noEmit` passes for `server` and `ui/src`. `pnpm check:token-gates` is clean. ## Risks - A task Retry during a scheduled retry now starts the run immediately, through the existing "Retry now" promotion gates (pause, budget, cleanup waits). A gate that holds it is reported, not bypassed. - "Retry now" is stricter: a member without read or change access to the task gets 404 or 403, as on `/retry`. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fd2e7482c0 |
fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop
## Linked Issues or Issue Description
Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).
**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.
**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.
**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
e015c2ef21 |
fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path - #20145 prepares the next Claude provider generation at park, and #20190 restarts the ACPX provider when a turn settles. They move the sidecar respawn off the follow-up's critical path. - When that turn-end restart fails, the runner "fails closed": `restart_session_after_turn` saves the session `closed` and drops it. Within the run, the next turn fails exactly like a failed restart at turn start. That's intended. - Across runs it breaks the conversation: - the next run's `run.attach` refuses a `closed` session ("run.attach requires the same settled ACPX provider profile and session"); - it reconnects for about 10 s; - the controller then records a continuity break and starts a fresh Claude conversation (`run.startup.reuse provider_session: fresh/created`). - A local A/B (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`, on a host with load average about 17–27) confirmed it: | | |
||
|
Dotta
|
5c73633a8f |
fix(workspaces): preserve remote recovery holds during online import (#20298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace import preserves old controller-local files while unrelated remote work continues. > - A task can have an old local checkout and a separate durable remote workspace. > - A terminal remote run can retain a file-recovery hold on that task. > - The legacy source checker rejects remote references, so that hold blocks independent local imports. > - This pull request proves that the hold belongs only to the separate remote source and fences its finalizer during import. > - Old files can migrate while the remote recovery obligation stays intact. ## Linked Issues or Issue Description Refs #19873 and #19854 for the existing cohort and source-verification paths. Searched related online migration and remote recovery PRs; no duplicate fix was found. **What happened?** Online import rejects a legacy Git checkout with `online_workspace_source_scope_unproven` when a same-task failed run has a prepared reference to a separate durable remote workspace. The legacy run reader rejects remote references before the hold can be classified. **Expected behavior** Independent local files can import under their source fences. The separate remote workspace, dirty writer, allocation, accepted result and recovery hold must stay intact. **Steps to reproduce** 1. Create related local Git workspaces. 2. Record a terminal failed native run and an exhausted file-recovery hold for a separate durable remote workspace on the same task. 3. Import the local leaf online. The old checker refuses before capture. **Paperclip version or commit** Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`. **Deployment mode** Managed server with PostgreSQL and remote Runner execution. ## What Changed - Add a distinct remote-only preservation proof. Keep remote runs excluded from legacy export sources. - Use the canonical source inspector and prepared-reference matcher. Bind the run, finalizer, action, task, accepted result, authority, allocation and prior committed snapshot. - Acquire the held run's finalizer fence through the existing sorted lock path. Existing source-proof reinspection rejects changes before publication. - Preserve previous-release prepared references with an older baseline snapshot. Do not require that snapshot's epoch or source run to equal the current run. - Add real PostgreSQL import, authenticated cold restore, preservation, refusal, lock-contention and publication-race regressions. Document the behavior. ## Verification - Genuine baseline failure on unchanged product: the real importer refuses with `online_workspace_source_scope_unproven`. - Final focused regression file: 23 tests passed in 13.19 seconds through the canonical isolated PostgreSQL wrapper on Node 24.22.2. - 24 existing affected migration/recovery tests passed against the same final production bytes. Another 83 tests were explicitly excluded by the focused selection; no full-suite pass is claimed. - The positive imports local bytes, confirms the remote finalizer fence is held during upload, leaves all remote records unchanged, parks the original repository and authenticates/restores the archive. - Refusal and race cases cover malformed, foreign, same-local and mixed references; active owners/writers; changed allocation, result, task, action, authority and baseline; and finalizer contention. - Server TypeScript check, module-boundary check and `git diff --check` passed. - An intermediate run had one incorrect assertion that expected an explicit empty legacy-source array. The canonical journal omits that empty field. The final assertion accepts absence and still forbids any exported remote source. Final relevant results are 23 new plus 24 existing passes, rather than a claim that the earlier invocation passed in full. - Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`: 55 successful checks and 4 intentional skips. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42650025033). Greptile reviewed that head at 5/5 with no actionable findings or open inline/review comments. Hosted continuation remains pending normal delivery. ## Risks - A false disjointness proof could admit an unsafe local import. Strict company/task/source binding, the canonical inspector, finalizer fencing and publication revalidation retain refusal on uncertainty. - This does not recover or settle remote files. Dirty authority and the original recovery obligation remain unchanged. - No schema, public API, archive format or saved-run identity changes. Existing persisted state carries forward. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
d10c6e0e8b |
fix(chat): hold a chat message at the agent's daily limit instead of dropping it (#20292)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat sends each person's message to the conversation agent through a durable outbox (`deliverConversationComments`). The outbox keys each wake receipt as `conversation-comment:<commentId>`. > - The outbox treats any receipt with that key as delivered. Queue admission writes a `skipped` receipt when the agent reached its daily run or cost limit. > - So a chat message sent after the agent used its daily allowance was consumed silently. No run started, the chat showed nothing, and nothing resumed it. > - A person's message must never be silently dropped. The daily limit is an operator control, so the message must wait honestly, not bypass the limit. > - This pull request makes the outbox hold the message at the daily limit, like it already does for a paused agent, and resume it when the limit resets. > - The benefit is that a chat message at the daily limit is saved, shown with its reason, and sent automatically at 00:00 UTC. ## Linked Issues or Issue Description No public issue exists. Inline bug report: **What happened** In a chat eval, the conversation agent (CEO) had `maxDailyRuns: 3`. After three chat turns and a separate failed-model phase on another agent, the person sent a fourth message. The comment was saved (POST 201), but no CEO run started. The conversation stayed `active` with no `executionRunId`. The ownership reconciler later reported `I7_visible_next_action`. The server log shows no wake error. Queue admission had written a `skipped` receipt (`heartbeat.daily_run_limit`) under the outbox key, so the outbox counted the message as delivered. **Expected behavior** A chat message that reaches the agent's daily limit is not consumed. It stays saved, the chat shows why it waits, and it is sent automatically when the limit resets. The daily limit still applies: the message does not start a run early. **Steps to reproduce** 1. Set a conversation agent's `runtimeConfig.heartbeat.maxDailyRuns` to N. 2. Send N chat messages and let each turn finish. 3. Send one more chat message. 4. See that no run starts, the chat shows no waiting state, and the message is never answered (a `skipped` wake receipt with reason `heartbeat.daily_run_limit` exists for `conversation-comment:<id>`). ## What Changed - `server/src/services/agent-conversations.ts`: add `conversationDeliveryHold`, which applies the same daily run/cost limit check as queue admission. The outbox returns `deferred` at the limit and writes no receipt, so the message stays owed. It reports the next UTC day start through a new optional `onDeferredDeadline` option. - `server/src/services/recovery/service.ts`: stranded-work recovery passes its `onDeferredDeadline` to the outbox, so the held message is retried when the limit resets. - `server/src/services/heartbeat/queue.ts`: if the limit is reached while an outbox wake is in admission (a race), admission throws a 409 with `reason: heartbeat.daily_*` for `conversation-comment:` keys instead of writing a `skipped` receipt. The outbox already maps a matching 409 to `deferred`. Other wakes keep the current skip behavior. - `server/src/routes/issues.ts` (`GET /issues/:id/queued-comments`): when a chat has no queue receipt but has messages held by the daily limit, return them as a `deferred` queue with `executionWait` (reason and message). Edit, send and discard stay off because the outbox owns delivery. - `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: show the `executionWait` message for such an outbox-held queue (no queue id and no target run). Receipt-backed queues keep the current behavior and do not duplicate the task status. - Tests: three real-database tests in `agent-conversation-delivery.test.ts` (hold and resume after the limit resets, recovery deadline, and a limit reached during real `heartbeatService.wakeup` admission) and one UI test. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversation-delivery.test.ts`: 10 passed. Without the source change, the three new tests fail (`expected 'delivered' to be 'deferred'`; the real queue path writes the skipped receipt). - `cd server && pnpm exec vitest run src/__tests__/issue-recovery-conversations.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/heartbeat-undispatched-execution-wait.test.ts src/services/heartbeat/queue.test.ts src/__tests__/agent-chat-messages.test.ts`: 215 passed. - `cd ui && pnpm exec vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx`: 36 passed. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: clean. `node scripts/check-token-gates.mjs`: clean. - Manual: give a chat agent `maxDailyRuns: 1`, send two messages. The second message shows in the queued messages bar with "<agent> reached its daily run limit. Your message is saved and sends automatically when the limit resets at 00:00 UTC." ## Risks - Behavior change for chat only: a chat message at the daily limit now waits in the outbox instead of being consumed. It is sent after 00:00 UTC. If an operator raises the limit, the message is sent on the next recovery pass for that chat; the UI message does not promise an earlier send. - Saved state from the previous release: no persisted shape changes. Messages that an earlier release already consumed with a `skipped` receipt stay consumed. They are not replayed later, which avoids sending old messages without notice. - The queued-comments read adds one agent read and one daily-limit count for a chat with no queue receipt. This is the same query that admission runs. - The recovery deadline is in-process; instance-sleep scheduling does not get a new deadline from this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
87e755ce8e |
fix(recovery): verify cross-task warm sandbox handoffs and say plainly when recovery refuses (#20282)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes keep a warm sandbox and hand it to
the next run through a chain of reusable leases
> - Acquisition binds that chain to the execution workspace, so a task
can inherit a warm sandbox from an earlier task in the same workspace
> - Saved-files recovery verifies the same chain, but it required every
earlier lease, run, and finalization to belong to the recovering task
> - A valid chain that started with another task therefore always failed
with `allocation_ownership_ambiguous`. That code is `service_required`,
so recovery stopped after one attempt, and each board Retry repeated the
same refusal
> - This pull request makes the chain check match the acquisition rule,
and it checks each predecessor against its own task
> - It also makes the board say plainly when recovery refuses the saved
state, because "failed after automatic retries" was not true and Retry
looked like it did nothing
> - The benefit is that a task blocked with "Saving the agent's files
failed after automatic retries" can recover its saved files, and a
refusal that Retry cannot change is shown with its reason
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn, and its result was
accepted. The copy of its files back to the host failed and was
scheduled to retry. A graceful server restart then interrupted the
retry, so the run settled with
`native_workspace_sync_out_retry_exhausted`, and the task became
blocked. Every saved-files recovery generation then failed on its first
attempt with `allocation_ownership_ambiguous` (`service_required`). The
board showed "Saving the agent's files failed after automatic retries.
Retry to try again". Each Retry replaced the generation
(`issue.recovery_generation_replaced`), and the new generation failed in
the same way within one second. To the user, Retry did nothing. The
sandbox lease chain was valid: same agent, same execution workspace,
same sandbox, and matching handoff receipts on all eight leases. But its
first three leases belonged to an earlier task in the same workspace.
**Expected behavior**
Recovery must accept a warm sandbox chain that acquisition accepted.
Acquisition binds reuse to the execution workspace, and it requires the
same task only when the lease has no workspace. Recovery must check each
predecessor run and finalization against that predecessor's own task. It
must still refuse a chain whose records disagree or that has no shared
workspace.
**Steps to reproduce**
1. Run a native task in a remote sandbox with `reuse_by_environment`,
with an execution workspace that another task also uses.
2. Run the other task in the same workspace first, so the sandbox lease
is handed from that task to this one.
3. Make the last run of the second task fail to copy its files back (for
example, restart the server during the copy-back retry).
4. Wait for saved-files recovery, or click Retry. Recovery fails with
`allocation_ownership_ambiguous`, and Retry repeats the failure.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
6958cb112c |
fix(ui): keep responder avatars visible across stale execution outcomes (#20293)
## Thinking Path > - Paperclip lets people manage agents and send follow-up messages in task chat. > - The responder identity should stay visible while a message starts a run. > - Comment storage, task reads, and run discovery finish separately. > - An older intent can resolve during admission of the new message. > - A reopened task can also retain its previous terminal status until the next read. > - This change keeps those older outcomes from clearing the new startup feedback. > - The avatar stays visible until the new run or its own admission outcome arrives. ## Linked Issues or Issue Description Related fixes: #17885 and #18312. A search found no other PR for these races. **What happened?** After Send, the agent avatar can disappear before the new run appears. A prior intent changing from working to resolved clears the new submission. Sending a follow-up that reopens a done or cancelled task also clears startup feedback against the old task status. **Expected behavior** Keep the responder visible through comment acknowledgement and run discovery. Stop startup feedback on a current failure, blocker, or completion. **Steps to reproduce** 1. Send a follow-up while a previous intent is still reported as working. 2. Let that same intent resolve before the new run read completes. 3. Observe the responder disappear. Alternatively, send a follow-up to a done or cancelled task and delay the reopened task read. **Paperclip version** Reproduced on source master at `33a23e0246`. Rebased onto `3ef45fdd92` before submission. **Deployment mode** Local component tests reproduce the read ordering. An isolated local trusted instance verifies the real Send-to-process-run path in Chromium. It does not invoke a live model provider. No staging message was sent. ## What Changed - Record the previous intent identity. Its later resolution cannot acknowledge the new message unless the observed run belongs to that message. - Pass the existing reopen request into shared submission state on both chat surfaces. - Keep startup feedback through the previous terminal status. Restore ordinary completion handling once the reopened status arrives. - Add six Send regression cases, a live-run handoff assertion, and coverage for later completion and ordinary terminal notes. - Clarify identity continuity in DESIGN.md. ## Verification - Red: all six new Send cases fail on unchanged master. - Green after rebase: 531 tests pass, one existing test is skipped, across TaskChatThread, IssueChatThread, and useRunnerSubmission. - Chromium: the existing compact-startup responder test passes on a disposable local instance. Frame sampling checks that the same identity stays mounted through acknowledgement and process startup. - UI typecheck and token gates pass. Initial typecheck lacked generated plugin SDK declarations; it passed after local server setup built that dependency. - Affected checks passed after rebase on `de9ed52c0b`. Final CI on `de9ed52c0bd6a6f7738614af31aec1eae2b871d0`: 59 checks passed, four skipped, no failures. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42646038090). Greptile and Paperclip review both gave 5/5 with no actionable findings or unresolved review threads. ## Risks Low scope: client startup feedback only. An old intent must not hide a new admission, but a new failure must still end it. Existing backoff and failure tests remain covered. The reopen exception applies only to an explicit reopen and ends when the reopened status arrives. No API, database, permission, or provider behavior changes. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed in this session. Capabilities used: reasoning, source editing, shell execution, and browser verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
052c638b51 |
fix(environments): honor stopped-task proof after warm preparation (#20289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A warm plugin Runner retains its task for later runs. > - A follow-up can find that the retained task has stopped. > - The resume path already asks settle to check status before it sends stop. > - A saved startup marker disabled this order, even after warm preparation had finished. > - This pull request applies the same stop-proof check to all observed-stop releases. > - The next run skips a repeated stop when status proves both execution and required preparation have stopped. ## Linked Issues or Issue Description Fixes: #20224 Refs #20203 ## What Changed - Remove the startup-marker exclusion from the existing status-first path. Keep the existing shared confirmation check. It still requires `preparationStopped: true` for startup work. - Keep stop followed by status when the first status lacks the required proof. Keep the startup drain, credential writeback, complete, and cleanup failure handling. - Add retained-task coverage with the startup metadata that warm preparation saves. Cover a startup cleanup whose first status lacks preparation proof. - Update the plugin task lifecycle contract. ## Verification - Local macOS, source base `3e8744fa91`: from `server`, `pnpm exec vitest run src/__tests__/environment-task-runtime.test.ts src/__tests__/plugin-warm-tasks.integration.test.ts src/__tests__/plugin-task-credential-writeback.test.ts` passed: 3 files, 82 tests. The integration tests use real embedded PostgreSQL and controlled plugin results. - After restoring the fix, the two affected files passed again: 73 tests. - Independent parent verification on commit `19f720ff68`: the two files filtered to the changed startup-proof and retained-task cases passed (4 tests). Independent source review found no actionable issue. - Baseline proof: with only the source condition restored to upstream, the new retained-task test with saved startup metadata fails because it sees an extra `stop`. The case without that metadata passes. The source fix was then restored. - The retained-task regression asserts exactly `environmentResumeLease`, `status`, `complete`, and `environmentAcquireLease`, with both task operations bound to the old task. The startup test also proves that missing preparation proof sends `stop` and a second `status`. - Test setup: `node scripts/agent-preflight.mjs --skip-rust-debug --explain` installed dependencies, then stopped because Cargo is unavailable on this Mac. Initial test collection failed because the plugin SDK output was absent. `pnpm --filter @paperclipai/plugin-sdk build` built that required dependency and the shared package; the targeted test run then passed. - Standard Greptile reviewed commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a` at 5/5 with no review threads. [Review](https://github.com/paperclipai/paperclip/pull/20289#issuecomment-6743829614). - Full local checks were not run, as requested. [GitHub PR workflow 42644023868](https://github.com/paperclipai/paperclip/actions/runs/42644023868) passed on final commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a`. It supplied broad typecheck, build, and test coverage. The separate Snyk status passed. Optional Storybook checks were skipped. - The change removes one provider stop call on the reported path. The issue's measured eight calls would become seven if other calls stay the same. The earlier five-call target is not claimed. No deployed latency measurement is claimed. ## Risks - A provider that cannot prove preparation stopped still gets stop and status. That fallback can cost one extra status call, as the existing status-first contract already allows. - A status failure keeps cleanup pending and blocks replacement. No termination proof or credential handoff is skipped. - No schema, API, dependency, or timer change. No added work on successful task reuse or ordinary task dispatch. - Live provider acceptance remains unrun. The local evidence checks orchestration and stop-proof handling. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository inspection, shell tools, and code execution. The exact serving model ID and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
09412e652a |
fix(runner): allow warm release to drain after provider restart (#20290)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm Runner sessions park after a turn so follow-up tasks can claim the same process. > - Claude restarts its ACPX provider after the controller receives the terminal result. > - That restart can delay the drain receipt beyond the release path's one-second limit. > - A failed drain prevents parking, so the idle timer later closes the retained Runner. > - This pull request uses the existing warm admission budget for release drain. > - The benefit is retained Runner reuse without moving provider restart onto task dispatch. ## Linked Issues or Issue Description Fixes: #20225. Refs #20190 and #20145. Related performance follow-up: #20224. ## What Changed - Align the release drain timeout with the existing 30-second warm admission budget for every provider. Close paths keep their explicit remaining deadlines. - Keep the existing immediate completion when the drain receipt proves settlement. Add no sleep, retry, provider branch or dispatch work. - Add a real Runner regression with a fake ACPX sidecar whose replacement start takes 1.5 seconds. One release call must park the Runner, then a follow-up must claim the same Runner process. - Document the release drain budget. ## Verification - Targeted command in `packages/paperclip-runner`: `vitest run src/live/runnerd-codex-transport-release-resume.test.ts src/live/runnerd-codex-transport-settlement.test.ts src/live/runnerd-codex-transport-deadlines.test.ts`: **46 passed** across 3 files, 157.78 seconds. - Independent parent verification reran the slow-restart and ordinary cached cases: **2 passed**. Source review confirmed that release is the only caller using the changed default; close callers supply explicit deadlines. - Negative control: with the original one-second timeout, the new test fails because the single release call returns `false`. The fixed code passes the new case. - Built only the current-source Runner and two fixture binaries required by these tests. The first sandboxed invocation could not bind its loopback listener (`EPERM`); the test ran with loopback access afterward. - Full GitHub PR workflow [42644210993](https://github.com/paperclipai/paperclip/actions/runs/42644210993) passed on head `9cfda0927b514d8d40443b612afa92525b35b700`. Standard Greptile scored this same head 5/5 with no review threads. Full repository checks ran in GitHub CI, per the requested local test scope. No live Claude session or hosted deployment was tested. ## Risks - An unresponsive Runner can now hold release drain for up to the existing 30-second warm admission limit. A healthy Runner returns as soon as its receipt arrives. The provider restart stays at turn end; task dispatch gets no fixed delay. - Saved state, protocol and provider identity formats do not change. Previous-release state carries forward under the existing validation. - The controlled fake-sidecar test proves the restart timing race and process reuse. It does not qualify live provider latency or a hosted rollout. ## Model Used - OpenAI GPT-6 through Codex, with repository tools, code editing and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
dc9ad17515 |
perf(storage): time workspace snapshot save and restore steps and count storage retries (#20280)
## Thinking Path
- Cloud durable storage is a major source of failures and slowness on
staging. For example, a staging task failed when saving files after
automatic retries.
- Before measuring or fixing it, we need to see where storage time goes.
Today a saved-workspace restore before the turn appears only inside the
coarse `workspace_environment_realize` phase.
- After the turn, the save's encrypt, put, verify and multipart throttle
backoff have no timing at all. Only `captureMs`, `transferMs`,
`verifyMs`, `uploadMs` and `publishMs` exist, in `saveTiming`.
- Storage retries happen in the SDK and in the multipart uploader,
invisibly.
- This adds the measurement first, so the storage-mode chaos waves
(MinIO behind a fault proxy) can report save and restore p50/p95 and
point at the dominant wait.
## Linked Issues or Issue Description
**What happened?**
Durable-storage saves and restores were opaque in the run log and save
timing. A slow or repeatedly retried storage call could not be told
apart from slow capture, encryption or sandbox work.
**Steps to reproduce**
1. Run a remote task with S3-compatible durable storage, so it saves
after the turn and restores on the next run.
2. Inspect the run log and the `remote_workspace_snapshot` operation's
`saveTiming`. The restore's download, decrypt and verify, and the save's
put, verify and throttle backoff, have no timing.
**Expected behavior**
Each storage step has a duration, and storage request retries and
backoff are counted.
## What Changed
- `server/src/storage/request-stats.ts` (new): an async-local collector
of storage request statistics. Counts only:
- `requests`, `attempts` and `failedRequests`;
- `retryDelayMs` (SDK waits);
- `parts`, `throttled` and `backoffMs` (the multipart uploader's waits).
- `server/src/storage/s3-failure.ts`: `withS3FailureDiagnostics`, which
wraps every S3 send, records each request's SDK `$metadata.attempts` and
`totalRetryDelay`, on success and on failure.
- `server/src/storage/s3-multipart.ts`: records each uploaded part and
each resend wait, flagged throttled when the backend returned 503, 429
or SlowDown.
- `server/src/services/remote-workspace-snapshot-storage.ts`:
- Each step is a `storage.snapshot.*` run phase:
- save: `prior_attempts`, `encrypt`, `digest`, `put`, `verify`;
- restore: `download`, `decrypt`, `verify_plaintext`, `publish`.
- A caller can pass a `timing` record that receives the step durations,
the bytes and the request statistics.
- `server/src/services/remote-workspaces.ts`:
- The save passes a timing record, kept in `saveTiming.storage`.
- The sandbox unpack during a restore is its own phase,
`workspace.snapshot.sandbox_restore`.
- `execution-preparation-timing.ts`: the new phase names.
- `doc/run-log-events.md`: a new "Durable storage timing" section.
Because the restore runs inside run preparation, its steps of 50 ms or
more become `run.performance.span` rows with no extra plumbing. This is
run-log and instance-database data only; there's no Telemetry change.
## Verification
- `server`: `storage-s3-multipart-backoff`, `storage-s3-multipart`,
`storage-s3-multipart-throttle`, `storage-s3-failure-diagnostics`,
`remote-workspace-snapshot-storage` and `execution-preparation-timing`
tests: 86 passed. They include:
- collected stats for a throttled multipart upload: 12 requests, 7
failed, 7 throttled, 35.5 s backoff and 3 parts;
- a save and restore that fill every step duration and the byte count.
- `remote-workspaces.integration.test.ts`: 112 passed. Its `saveTiming`
assertion now includes `storage`.
- Server typecheck: no errors in the touched files.
## Risks
- **Overhead:** one async-local lookup per S3 request, plus timestamps
per snapshot step.
- **Collector scope:** the collector only exists inside a snapshot save
or restore. Other storage callers are unaffected; they record nothing.
- **`saveTiming`** gains a `storage` object. Readers that compare it
exactly (one test, updated) will see the new field.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
3ef45fdd92 |
fix(recovery): let a new task wait for its manually paused assignee (#20281)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A person can pause an agent. Tasks assigned to a paused agent wait until the person resumes it. > - Stranded-task recovery looks for assigned work that cannot continue and moves it to `blocked` for the board. > - A task created for a manually paused agent has no run yet. Recovery saw an assignee that is not invokable and moved the new task to `blocked` a few seconds after creation. > - The person paused the agent on purpose. The blocked task and its "the board must choose the next action" comment fight that decision, and the person must unblock the task by hand after resume. > - This pull request lets new `todo` work with no run wait while a manual pause lasts. The existing assigned-todo dispatch starts it when the agent is resumed. > - The benefit is that work queued for a paused agent starts on resume, as the pause promises. ## Linked Issues or Issue Description **What happened** A chat evaluation pauses an agent (Morgan) and then runs `paperclipai issue create ... --assignee-agent-id <Morgan> --status todo`. The CLI returns the task with status `todo`. Within about five seconds the task changes to `blocked`, with the system comment "Paperclip cannot safely continue automatic recovery because the original assignee is not invokable. The source assignment is unchanged and the board must choose the next action." The server log shows only the expected 409 "Agent is not invokable" for the assignment wake. **Expected behavior** The task stays `todo` while the person keeps the agent paused. When the person resumes the agent, the task starts once. A comment and a `blocked` status are not added. **Steps to reproduce** 1. Pause an agent from its page. 2. Create a `todo` task assigned to that agent. 3. Wait for the stranded-task recovery sweep. See that the task is `blocked` with the comment above. ## What Changed - `server/src/services/recovery/service.ts`: in `reconcileStrandedAssignedIssues`, skip a `todo` task with no run when its assignee is paused with a `manual` lifecycle hold. The existing rule above it already skips work whose run the pause cancelled. Other pause causes, such as a budget hold, keep the existing escalation. - `server/src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: a PostgreSQL test. The task stays `todo` with no comment and no run during two recovery sweeps. After resume it starts exactly once and finishes. A budget-held assignee still escalates. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 2 passed. Without the fix, the first test fails with `expected 'blocked' to be 'todo'`. - `npx vitest run src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-refused-assignment-wake.test.ts`: passed. - `src/__tests__/issue-tree-pause-resume-run.test.ts` had timing failures on this low-memory machine both with and without this change (different tests each time). Those tests do not pause agents, so this change does not reach them. CI runs them on this branch. ## Risks - Low risk. The skip needs all of these: the agent is paused, it has a `manual` hold, the task is `todo`, and the task has no run. Work that already ran keeps the current rules. - No persisted state changes shape. State saved by the previous release carries forward: a task that the previous release already moved to `blocked` stays `blocked`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20143 (a pause's stop never blocks the task behind an incident), #9998 (open: reject paused agents as assignment targets). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6db0bb9776 |
fix(chat): tell agents which CLI commands a person runs in their own terminal (#20267)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The chat guidance in `server/src/services/agent-conversations.ts` tells every chat agent how to answer. > - People often ask a chat agent for a terminal command: create a task, download a plan, inspect or stop a run, or point the CLI at a company. > - The guidance did not name the Paperclip CLI. Agents guessed: they did the work themselves, wrote `curl`, used placeholder company ids, or invented an API address. > - The person then has to fix the command, or gets a file in the agent's workspace instead of on their own machine. > - This pull request adds one guidance paragraph. It names the supported `npx paperclipai` commands, asks for real ids from the conversation, and asks for the person's own Paperclip URL as a placeholder. > - The benefit is a command the person can run as it is, with the right company and the right actor. ## Linked Issues or Issue Description **What happened** Three chat evaluations (cli-operations) asked for terminal commands. "Show me how to point my terminal at the Acorn company." got `context set` with `<acorn-company-id>` and an invented `http://localhost:3100`, although the agent can read its company id. "Download this task's plan from the terminal as a file." got a file saved in the agent's own workspace and an attachment link, with no terminal command. "Show me the command to inspect Morgan's last run and stop it if it is still running." got `curl` with environment variables that exist only in the agent's runtime. **Expected behavior** The agent gives `npx paperclipai` commands that the person runs in their own terminal. The commands use real ids that the agent can read in the conversation. The Paperclip address is the placeholder `<your Paperclip URL>`. The agent says which commands need a board user. **Steps to reproduce** 1. Open a chat with an agent. 2. Send "Download this task's plan from the terminal as a file." 3. See that the agent saves the file itself and gives no terminal command. ## What Changed - `server/src/services/agent-conversations.ts`: add a chat guidance paragraph for terminal and CLI requests. It names the supported commands for task creation, document download, run read and cancel, and context profiles. It asks for one command per line, real ids, and the `<your Paperclip URL>` placeholder. - `cli/src/__tests__/chat-cli-guidance.test.ts`: a new test reads the guidance and resolves every `npx paperclipai` command against the registered commander commands. It fails on an unknown command, flag or missing argument. - `server/src/__tests__/agent-conversations.test.ts`: check that the paragraph is in the directive and in the chat prompt. ## Verification - `cd cli && npx vitest run src/__tests__/chat-cli-guidance.test.ts`: 3 passed. - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts`: 80 passed. - The chat evaluations `cli-operations.choose-cli-context`, `export-document-cli` and `inspect-run-cli` run with this commit on an evaluation branch. Results are added in a comment. ## Risks - Low risk. The change adds prompt text and tests. Routes, permissions and responses do not change. - The chat prompt is about 300 words longer. - If a CLI command or flag changes, the new CLI test fails until the guidance changes too. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #20029 (terminal sign-in guidance in the API catalog). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f7c85bca18 |
test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness boots a full temporary Paperclip instance per wave: embedded Postgres, workspaces and caches, often 5–13 GB > - At wave end it deleted that instance only when the wave passed with no harness error and no `--keep` > - Almost every chaos wave has at least one failing task, so almost every instance stayed on disk > - Over Nov 6–9, 359 `paperclip-runner-e2e-*` directories filled the machine's disk (ENOSPC), which stopped a running storm wave > - This pull request deletes the instance at the end of every wave, smoke and baseline run unless it is explicitly kept or its processes could not be stopped, and expires kept instances after 6 hours > - The benefit is that chaos waves can run back to back without filling the disk ## Linked Issues or Issue Description **What happened?** `main.ts` kept the temporary root whenever `keep || anyFailure || harnessError`. A failing task is the normal output of a chaos wave, so instances accumulated until the disk was full. The codex K13 storm w-20261109T113925-s9403 died after about a minute with every task cut off. **Steps to reproduce** 1. Run any wave in which at least one task fails. 2. The `paperclip-runner-e2e-*` directory under the OS temp dir stays behind, with no expiry. **Expected behavior** The instance is removed when the wave ends. Evidence is already in `results/`. Only an explicit `--keep`, or a stop failure that may leave processes running, keeps it, and kept instances expire. **Actual behavior** Every failing wave leaks its instance permanently. ## What Changed - New shared `tests/runner-e2e/instance-retention.ts`: - `releaseInstance` deletes the root unless `--keep` was passed or the processes could not be stopped. In those cases it writes a `.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets the directory to 0700. An optional `retain` list keeps small evidence files and removes everything else. - `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*` directories that carry the marker, are older than 6 hours, and have no live `postmaster.pid` anywhere inside them, and that no live process command line references. Each root is made removable first, and one failed delete does not stop the sweep. Unmarked directories are never touched, so existing directories from older runs stay until someone decides on them. `recovery_state` markers never expire. - `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets it. - Callers: - A wave whose Daytona cleanup is still pending (environment DELETE blocked, sandboxes still running after cleanup, or cleanup errors) keeps its instance as `recovery_state`, because its database owns that cleanup. - The wave releases its instance before writing the ledger and summary, so a failed report write cannot skip cleanup. - Restart chaos `main.ts` (crash-point suite included) releases the instance in its final cleanup, which also runs on failure and interrupts. The crash-point task-count check moved above supervisor creation so that error no longer leaks a fresh root. - `ttft-baseline.ts` releases after stopping the supervisor, including on SIGINT and SIGTERM. - The rolling-deployment and controller-replacement smokes now delete their roots, keeping only the evidence report, logs and screenshots. If cleanup is unconfirmed, the root is kept with a marker. - `launch.ts` marks its unconfirmed-remote-cleanup root as `recovery_state`, which never expires, and its `PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing deletion path is unchanged, and the file gets one line shorter. - README: the instance lifecycle and `--keep`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328 files, 6915 passed. The new retention tests cover: - deleting a finished instance, and retaining only listed evidence; - keeping it with a marker and mode 0700; - sweeping only marked, expired instances whose Postgres isn't running, while sparing fresh, live, recovery-state, unmarked and unrelated directories; - a stale `postmaster.pid`; - sweeping on `createRunnerE2ETemporaryRoot`. The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran the sweep. - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221 passed. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and `tests/restart-chaos`: clean. - No wave or smoke was run, because the disk is still nearly full. ## Risks - A failed wave's instance is no longer available for post-mortem DB inspection unless the wave ran with `--keep`. Evidence bundles in `results/` are unchanged. - The sweep relies on `postmaster.pid`. An instance whose Postgres was started outside the harness, without that file, could be removed after 6 hours. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Pavel-GuseynovandClaude Opus 5.5
|
4d297b998a |
fix(server): give remote MCP tool calls a 30 second default deadline and record stream failure evidence (#19768)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The tool gateway calls remote MCP servers over Streamable HTTP, and a server can answer `tools/call` with a `text/event-stream` body > - Such a server can send progress notifications before the response, and a slow tool can take more than 10 seconds > - The gateway aborts a remote MCP `tools/call` that uses the default deadline after 10 seconds, the general tool default > - When the call fails, the call evidence does not show how far the stream got, or whether the cause was the deadline or a broken transport > - This pull request gives remote MCP calls a default deadline of 30 seconds, the default response timeout of the remote HTTP client, and records the failure cause and the stream progress > - The benefit is that slow remote tools complete, and an operator can tell a deadline from a peer reset in the call evidence ## Linked Issues or Issue Description Refs #10147. That issue reports that a named MCP gateway aborts remote tools after 10 seconds and ignores the connection timeoutMs. This pull request raises the default deadline of a remote MCP `tools/call` to 30 seconds. It does not make the named gateway read the connection timeoutMs. Related: #11910 (open) keeps a connection callable after one timed-out tools/call. On current master a timeout and a JSON-RPC response error already keep the connection healthy, and this pull request keeps that policy. I searched pull requests and issues for "event stream tools/call", "SSE MCP tool call hang", "remote MCP timeout stream", "mcp text/event-stream" and "tool_timeout remote MCP". The only related open pull request is #11910. ## What Changed - `server/src/services/remote-http-fetch.ts`: the default response timeout is exported as `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` (30,000 ms, unchanged value). - `server/src/services/tool-gateway.ts`: - A remote MCP `tools/call` that uses the default deadline now gets `DEFAULT_REMOTE_HTTP_RESPONSE_TIMEOUT_MS` instead of the general tool default of 10 seconds. The Railway `run-command` budget and an explicit `timeoutMs` keep their values. - The call evidence records the response status, content type and request id before the body is read. A body that fails partway records the bytes received so far as its size. - A failed call records `failureKind` (`invocation_timeout`, `protocol_failure` or `transport_failure`), the deadline and whether it was aborted, the stream progress, and the error with its cause chain. Each string in the chain is redacted with `redactSensitiveText`, and a cyclic cause chain is cut with a "[Circular cause]" marker. - A deadline abort is a 504 `tool_timeout` and starts a fresh MCP session on the next call, as before. The connection health policy of master does not change. - A deliberate stop of a call (`elicitation_required`, `provider_interaction_required`, `oauth_insufficient_scope`, `oauth_refreshed_retry_required`) keeps its own reason when the deadline passes while it is recorded, as on master. An error that is already a gateway error also gets a failure kind: `tool_timeout` is `invocation_timeout`; a malformed, invalid or oversized response is `protocol_failure`; and `remote_http_*` (for example a DNS failure) and `mcp_remote_fetch_failed` are `transport_failure`. - A non-success response body is read as a stream, so a body that fails partway records the bytes that arrived. The reader stops at the size limit and cancels the body, as the `tools/call` reader does. #19785 makes the same reader change for the context path. - `server/src/services/mcp-http.ts`: `readMcpHttpResponse` takes an `onProgress` callback. It reports bytes, chunks and events received, whether the matching response arrived, and whether the stream ended. As on master, the read returns as soon as the response with the matching id arrives. - Tests: - `server/src/__tests__/mcp-http.test.ts`: a partial event followed by a peer error keeps the progress that was consumed, and releases the reader. - `server/src/__tests__/tool-gateway.test.ts`: six scenarios run against a remote server that streams one progress event first: slow success after 12 seconds, the default deadline, an explicit 1,500 ms deadline, a peer reset, a cyclic error cause, and a primitive error value. - `server/src/__tests__/tool-gateway.test.ts`: three regression tests: a tool question sent after the deadline passed keeps `elicitation_required`; a DNS failure records `transport_failure`; and a 503 body that fails partway records the bytes that arrived. All three fail oncanary/v2026.1109.0-canary.14 |
||
|
DottaandClaude Opus 5.5
|
3e8744fa91 |
perf(task-chat): stop a slow run transcript from hiding the chat (#20277)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat shows saved messages, run transcripts and the composer > - The chat hides the messages and the composer until every run transcript loads. This prevents layout movement > - On staging, one task's run-log read took 11–12 s. Saved messages and the composer stayed hidden for 13.9 s after a click, and for 14.6 s again on revisit > - This pull request limits how long a transcript can hold the first reveal after saved history is ready > - The benefit is that a slow run log no longer hides the conversation, and the layout contract stays in place ## Linked Issues or Issue Description **Describe the bug** When a task has a run whose transcript loads slowly, the whole chat stays behind the loading skeleton. The composer stays hidden too. On staging, a failed run with no saved reply needed an 11–12 s object-store log read. The chat appeared after 13.9 s. Saved messages that loaded in about 2 s were not visible during that time. **Steps to reproduce** 1. Open a task whose latest run failed before it saved a reply, and whose run log is large or slow to read. 2. Watch the chat area after the click. **Expected behavior** Saved messages and the composer appear when saved history is ready. Run output must not move the messages that are already visible. **Actual behavior** The chat stays hidden until the run log read finishes, or until the 15 s reveal timeout. Refs #20276 (the measurement harness and baseline plan). ## What Changed - `TaskChatThread` now waits at most `INITIAL_TRANSCRIPT_REVEAL_BUDGET_MS` (600 ms) for run transcripts. The time starts when saved history, the plan and chat metadata are ready. The budget is one deadline. A transcript that loads early does not restart it. - When the budget expires, the slow runs are latched for that task, so the decision does not change later: - **Live runs** stop holding the reveal. Their output continues to stream into the live tail at the bottom of the chat. - **Settled runs in compact history** stop loading. They use the existing fixed **More run output** control, so late output cannot insert rows above the reader. Selecting the control loads the full history, as it does today. - **Settled runs in other modes** (expanded history, conversation mode, a deep link to a comment) still wait, as before. In those modes, output would insert rows around saved replies. - **Live runs with a steering message or other timeline anchors** still wait, as before. Their output before the anchor goes into the main conversation above the anchored message. - A live run that finishes after the reveal keeps its visible transcript. Only runs that were already settled when the budget expired are deferred. ## Verification - `cd ui && npx vitest run src/components/TaskChatThread.test.tsx`. New tests for legacy and native runs: - A slow failed run in compact history: the chat reveals at the budget, the transcript is no longer requested, **More run output** appears, and expanding loads the run. - A slow live run: the chat reveals at the budget, the transcript keeps streaming, and the transcript stays after the run finishes. - A slow settled run without the compact control: the chat still waits. - The budget does not start while saved history is pending. - When one of two transcripts loads before the budget ends, the chat still reveals at the first deadline. - A slow live run with a steering message: the chat still waits. - A compact task with 300 slow failed runs: the chat reveals at the budget and defers every run. - `cd ui && npx vitest run src/components/task-chat/transcript-reveal-budget.test.ts`. The budget selection is one pass. A 5,000-run test counts anchor lookups and throws above one lookup for each live run. - Red check: with the budget timer disabled, the 6 new positive tests fail. With the change, they pass. - `npx vitest run src/components/TaskChatThread.test.tsx src/components/task-chat src/pages/IssueDetail`: 64 files and 1,609 tests pass. - `pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/task-loading-frame.spec.ts`: 7 of 7 pass. This covers a stable loading frame on desktop and mobile, composer geometry, the reading position, a stalled web font, and the Classic and mobile preference paths. - After deploy, staging is measured with the harness from #20276. The finish condition is that a slow run-log task opens in the same time as other tasks. ## Risks - After a slow transcript is deferred, a settled failed run in compact history shows its failure message and **More run output**, not its last transcript lines. The output is one click away. The server-side log read fix is a separate follow-up. - A live run's output can appear in the tail after the reveal. This is the normal streaming behaviour of the tail. - A 600 ms budget is a trade-off. Fast transcripts still arrive in the first frame, and slow ones no longer hide the chat. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ac0d0374f2 |
perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description **Problem or motivation** Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. **Proposed solution** Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. **Alternatives considered** Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. **Roadmap alignment** This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fb88a89f04 |
fix(chat): tell chat agents how to link a case page while Cases is enabled (#20265)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers questions about company records and links each record it cites > - Chat guidance gives link formats for tasks, approvals, runs and agent profiles > - Cases (an experiment) have their own page at `/<prefix>/cases/<identifier>`, but the guidance does not name it > - So an agent that answers a question about a case's fact history names the case but cannot link it > - This pull request adds a Cases directive to chat guidance, only while the Cases experiment is on > - The benefit is that a person can open the case, where its facts and audit history are, from the reply ## Linked Issues or Issue Description **What happened** In a chat eval with Cases enabled, a person asked who changed a case's diagnosis. The agent read the case audit and the task's document revisions and answered correctly. It wrote "case COM-C1" as plain text. It linked only the task. The guidance in the chat prompt has no link format for a case, and the agent does not guess routes. **Expected behavior** When a reply concerns a case (its facts, fact-change history or handoffs), the agent links the case page as a Markdown link, for example `[RUN-C1](/<prefix>/cases/RUN-C1)`, and also links the tasks it cites. When the Cases experiment is off, the prompt does not mention cases. **Steps to reproduce** 1. Enable the Cases experiment. 2. Create a case with a primary task, and change one of its facts. 3. In Agent Chat, ask "Who changed the diagnosis, and what did it say before?" 4. Before this change, the reply names the case identifier but has no link to the case page. ## What Changed - `server/src/services/agent-conversations.ts`: new `CASES_CHAT_DIRECTIVE`. It says what a case is, names the `/<prefix>/cases/<case identifier>` page, and asks for a Markdown link when a reply concerns a case. - `server/src/services/heartbeat/task-markdown.ts`: new `casesEnabled` input. A conversation prompt includes the directive under "Cases:" only when it is true. - `server/src/services/heartbeat/task-context.ts`: sets `casesEnabled` from the `enableCases` experimental setting, the same way as `statusCardsEnabled`. - `server/src/services/heartbeat/task-markdown.test.ts`: new test. The directive appears for a standing chat with Cases on. It does not appear with Cases off or for a non-chat task. The test also checks that the board route `cases/:caseIdentifier` exists in `ui/src/App.tsx`. ## Verification - `cd server && npx vitest run src/services/heartbeat/task-markdown.test.ts` passes (18 tests). - `npx tsc --noEmit -p server` passes. - The `cases.case-history` chat eval uses this guidance. Its rerun is on branch `chat-eval/case-unclear-9`. ## Risks - Low risk. The change adds prompt text only when the Cases experiment is on. It does not change any API or permission. - The prompt for chats on instances with Cases enabled is slightly longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M context, extended thinking, with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1004e55180 |
fix(recovery): stop stall wakes from failing setup on a held workspace (#20273)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, check for proof that its run would be refused. The proof is that the assignee's own latest run on the task failed setup with the unfinished-writer error, and the bound workspace hold still refuses a run. Then do not wake it; count it as deferred and re-check at most hourly. A different assignee (for example a controller-only agent that never uses the workspace), a later run, or a settled hold lets the wake through. Reservation already tried and failed the unchanged-writer release for that run, so the sweep does not repeat that proof. - `blocker-stall-wakeups.ts`: count failed runs that carry the stall (the keyed wake's run and keyless resource-wait retries that keep its context). Retry after 10, then 20 minutes, and stop after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `remote-workspace-writer-hold.ts` / `remote-workspaces.ts`: share the unfinished-writer error text as one constant. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 22/22. Added after review: a waiter with no refused run, or a reassigned waiter, is still woken; failed resource-wait retries count toward the limit. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. The parent's own earlier run failed setup on the hold, as on staging. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2dd7a98753 |
fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Recovery tells a waiting parent or dependent when a child or blocker stalls on the board (`issue_blocker_stalled`) > - A legacy child that an older release bound to its parent's remote workspace failed its save, and its save recovery was exhausted. Its ended run still holds the shared workspace, days past the one-hour holder cutoff > - Workspace reservation refuses a run on such a hold and fails setup with "The workspace still has an unfinished writer or save operation." Every stall wake of the parent created a run that could only fail this way > - The stall sweep treated each failed run as an undelivered wake and retried it every ten minutes, with no backoff and no limit. On staging this made 72 failed runs (58 of 60 failures since a deploy), in pairs every ten minutes > - This pull request makes the stall sweep hold a wake while the waiter's workspace hold refuses a run, and bounds retries of a wake whose run fails > - The benefit is that a held workspace no longer produces a failed run every ten minutes, and the wake still arrives once the hold settles ## Linked Issues or Issue Description **What happened?** A parent task was woken by `issue_blocker_stalled` twice every ten minutes. Each run failed in setup (`setup_failed`, `executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with "The workspace still has an unfinished writer or save operation." The parent's remote workspace was held by a child's ended writer run whose save recovery was exhausted (`native_workspace_sync_out_retry_exhausted`, last failure `agent_stop_unconfirmed`). The child shared the parent's workspace because an older release created it before subtasks got their own workspace (#19860). **Expected behavior** A wake that cannot start because its workspace is held must not create a failed run every time. It should wait until the hold settles. Automatic retries must be bounded. **Root cause** - `server/src/services/heartbeat/workspace-preparation.ts` rethrows `RemoteWorkspaceWriterActiveError` as a setup failure when the ended writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (documented, intended). - `server/src/services/recovery/blocker-stall-wakeups.ts` retried every non-delivered wake row (including `failed`) after `REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and it did not check whether the waiter could start at all. ## What Changed - `remote-workspace-writer-hold.ts`: add `remoteWorkspaceHoldRefusesStart`, the one rule for a hold that reservation refuses outright (unattributed writer, or a writer silent past the cutoff with no live owner fence). - `workspace-preparation.ts`: use that rule instead of an inline copy, so reservation and wake senders cannot drift. - `blocker-stall-wakeups.ts`: before waking a waiter, read its bound remote workspace hold. When the hold refuses a run, do not wake it; count it as deferred and re-check at most hourly. A workspace with an unchanged-release baseline is left to reservation, which can release it. - `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried after 10, then 20 minutes, and not after the third failed run. Refusals (wake-on-demand off, agent paused) keep the 10-minute retry. A new stall (new recovery action) wakes again. - `doc/execution-semantics.md`: document both rules. ## Verification - Red before the fix (`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new tests failed. The held-workspace test got a delivered wake; the backoff test got a third wake 11 minutes after the second failure. - Green after the fix: `npx vitest run src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20. - The new tests use the staging shape: a legacy `shared_workspace` remote workspace owned by the parent, a child bound to it, the child's failed native writer run ended four days ago with `native_workspace_sync_out_retry_exhausted`, a dirty authority, and the child's board-owned exhausted recovery action. They also prove that a writer inside the cutoff still gets its wake (the run parks in a workspace wait), and that the wake is delivered once the hold settles. - Related suites pass: `heartbeat-remote-workspace-busy.test.ts`, `heartbeat-dependency-scheduling.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `src/modules/run-dispatch`. - `npx tsc --noEmit -p server` passes. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input; required CI supplies the broad coverage. ## Risks - Low. No schema or persisted-shape change. Saved state from the previous release (existing wake rows, existing workspace authority) is read as-is: earlier `failed` stall wake rows count toward the new three-run limit, which stops the current loop at once after deploy. - A parent whose workspace stays held is not woken for the stall until the hold settles. The stall is still listed on its page and the child's recovery action stays with the board. - Not fixed here: the exhausted save hold itself is never settled automatically (`agent_stop_unconfirmed` exhaustion in workspace recovery), and legacy children that still share a parent's workspace are not moved. Both touch workspace recovery code that is under concurrent change and are left for a follow-up. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser read-only inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
610b54f551 |
test(chaos): close review-thread gaps in speed grading, S1 stops, readiness and stall reads (#20244)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades how real tasks survive control-plane restarts, and its grades decide which product fixes come next > - Reviewers left 46 unresolved threads on ten merged harness PRs (#20071, #20094, #20106, #20111, #20119, #20147, #20158, #20161, #20178, #20194) > - Several are real harness defects: false speed breaches from one-sample rows, a codex-only wave that cannot start, event paging that can loop forever, an S1 task failed for a Stop the harness never sent, and a readiness wait that aborts on a transient lifecycle error > - Wave S1 t05 (w-20261109T091247-s9806) hit the S1 defect: the run finished before the stop window, so neither the Stop nor the continue comment was sent, and the task was graded `done_too_late Infinity` > - This pull request fixes the valid findings and adds the missing tests > - The benefit is that a failed task points at the product, not at the harness ## Linked Issues or Issue Description Refs #20071, #20094, #20111, #20147, #20158, #20161, #20178, #20194 (review threads). The security finding from #20111 was fixed separately in #20228. **What happened?** Harness defects found in review and in waves: - Aggregate p50/p95 rows breached at n = 1–3, which caused most marginal speed failures. - `--adapters codex` with the default workloads failed at parse because M1 needs two adapters. - `runEvents` paged forever if a full page had no `seq`. - A timed-out task whose last sample was `done` was graded `done_too_late Infinity`, and could pass when a fault had no ready time. - S1 waited forever for a running run to stop; it also clicked Stop while another task's fault had the server down or recovering. - The readiness wait aborted on any `lifecycleError`, including the transient one lifecycle `defer` sets before a 60 s retry, and hid read errors as "not ready". Daytona used the 240 s local timeout. - One failed parent read could hide an M1 stall for the whole 10 s check window. ## What Changed - Speed: aggregate rows breach only from 5 samples (`insufficient sample (n < 5)` otherwise). Restart recoveries over the control-plane p95 budget now count in the per-run check, so a single slow recovery still fails. - Config: the default workloads skip M1 when fewer than two adapters are given; an explicit `--workloads M1` with one adapter still fails. - `runEvents` throws when a full page has no later `seq`; the run's speed read fails instead of hanging. - `gradeR1` fails a timed-out task as `no_rest_within_budget` before checking its status. - S1: the Stop waits for a ready server with no fault in progress. If the run already finished when the window opens, the harness records a missed Stop, skips the stop-triggered fault, still posts the continue comment, and the oracle checks only the continue. - Readiness: only a terminal lifecycle state (rejected, terminating, cleaning up, terminated) aborts at once. Any other lifecycle error is logged when it changes (`seed_agent_lifecycle_error`) and the wait continues until ready or the timeout, which names each agent's last lifecycle or read error. The agent API exposes neither `lifecycleOperation.retryRequired` nor a retry-claim timestamp, so a deferred retry cannot be told apart from a parked failure earlier. Daytona waits up to 10 min. - M1: a failed parent read retries on the next poll. - Child speed collection moved into `childSpeedTasksInTurn` so its one-at-a-time order is tested. - Tests for each change, plus the gaps reviewers named: the stop trigger's arming, stall-wake parsing, and `seedChaosLab`'s readiness wiring. - README: S1 missed Stop, R1 timeout rule, speed sample minimum. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 230 passed. Each new test failed before its fix, such as the paging test running out of memory in the loop and the stop-trigger test firing a fault after a missed Stop. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Wave evidence for the defects: S1 t05 in w-20261109T091247-s9806 (missed Stop); codex warm agent_boot p50 breach at n = 3 in w-20261109T104527-s9901. ## Risks - Aggregate rows with fewer than 5 samples no longer fail a wave. Per-run p95 checks still catch individual slow runs and recoveries, but a consistently slightly slow p50 in a small wave is now only reported. - A lifecycle error that persists without `retryRequired` now waits for the readiness timeout (4 min local, 10 min Daytona) instead of failing at once. The timeout message names the error. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1b09920a64 |
fix(issues): refuse agent status changes that detach another run's live execution (#20271)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that an agent run executes holds a checkout lock and an execution lock (`checkoutRunId`, `executionRunId`) > - A status change away from `in_progress` clears these locks. It does not stop the run that holds them > - Agents cannot stop another agent's run: run cancel and the task-tree pause are board-only > - But an agent that can edit a task (for example a lead with the manage-active-checkouts grant over its report) can change its status. The run then keeps running and is no longer bound to its task > - In chat evals, a lead "paused" a launch tree. It moved running subtasks to backlog. The runs kept running, detached > - This pull request refuses an agent's status change when a different queued or running run holds the task's lock. The denial is final, links the task, and names the person's controls > - The benefit is that agents cannot interrupt another agent's run through a status edit, and runs are not left orphaned ## Linked Issues or Issue Description Refs #20261 (chat guidance sends a task-tree pause to the person's Pause subtree control). Refs #19394 (resume restarts the work a subtree pause interrupted). **What happened** An agent changed the status of a task while another agent's run executed it. The request succeeded. The server cleared `checkoutRunId` and `executionRunId`. The run continued, but it was no longer bound to the task. In the chat evals `steering.stop-versus-pause` and `task-coordination.pause-tree`, a lead moved running subtasks to backlog to "pause" a launch. Their runs kept running, detached. **Expected behavior** An agent cannot get the effect of interrupting another agent's run. When a different live run holds the task's lock, the agent's status change is refused before any write. The refusal says it is final. It sends the person to the task and names the controls that really stop or hold the work: the run's Stop or Interrupt, or More task actions, then Pause subtree on the parent task. The lock-holding run, board users, and paths that stop the run themselves continue to work. **Steps to reproduce** 1. Create agent A and agent B, with B reporting to A (A gets `tasks:manage_active_checkouts` through the manager chain). 2. Give B an `in_progress` task. Set `checkoutRunId` and `executionRunId` to a `running` heartbeat run of B. 3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with `{ "status": "backlog" }`. 4. Before this change: `200`, the status is `backlog`, both lock fields are `null`, and B's run is still `running`. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it refuses a status change when `executionRunId` or `checkoutRunId` names a different run in the same company that is `queued` or `running`. It runs in `PATCH /issues/:id` after the final status and assignee are known (execution-policy transitions included) and before any run stop or write. It also runs in recovery resolution when the source status changes. - The guard applies to any status other than `in_progress`, also to the current status, because the service clears the locks for those too. - The write is conditional on the lock the guard checked. The route re-reads the lock and passes it to `svc.update` as `expectedExecutionLock`. The service compares it under the row lock and returns `409 issue_execution_lock_changed` if a checkout landed in between. - These cases stay allowed: the lock-holding run itself, board users, requests with no status, a change to `in_progress`, `cancelled` (the route cancels the run), reassignment (the route stops the run first), `done` through PATCH while a Runner goal is active (terminalization pauses the goal and stops the run; recovery resolution does not get this exception), and a lock holder that already finished. - `packages/shared/src/issue-write-denial.ts`: new denial code `issue_status_change_live_run` (409). The copy says the refusal is final, links the task as `[identifier](/prefix/issues/identifier)`, and names Stop or Interrupt and More task actions, then Pause subtree. A new optional `issuePath` context field carries the link. The UI notice renders it through the existing generic path. - `doc/SPEC-implementation.md`: documents the rule in the checkout contract section. - Paths checked with no change needed: there is no bulk issue update route; tree holds and run cancel are board-only; `POST /issues/:id/release` already lets only the assignee's checkout run release; API-catalog actions call the same PATCH route; MCP status changes already refuse with `MCP_ACTIVE_EXECUTION`. ## Verification - Red on master, green with this change: - `server/src/__tests__/issue-stale-execution-lock-routes.test.ts` (embedded Postgres, real issue service): a manager moves its report's running task to backlog. On master: `200`, status `backlog`, `checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409 issue_status_change_live_run`, row unchanged, run still `running`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued lock holder, and a repeated current status (red on master). Allowed cases: field edits, the lock-holding run, board, `cancelled` (run is cancelled), and finished lock holders; an allowed change carries `expectedExecutionLock`. - Same embedded-Postgres file: `svc.update` with an `expectedExecutionLock` that differs from the row returns `409 issue_execution_lock_changed` and leaves the row unchanged. - Also passed locally: `issue-closed-workspace-routes`, `issue-comment-reopen-routes`, `issue-recovery-actions`, `issue-scheduled-retry-routes`, `low-trust-red-team-routes`, `issue-update-comment-wakeup-routes`, `external-object-routes`, `packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`. - `tsc --noEmit` shows no errors in the changed files. Full typecheck, build and test are left to CI. ## Risks - Behavior change for agents: a lead or CEO that used status edits to take over a report's running task now gets a 409. The supported paths (comment, cancel, reassign, or the person's Stop or Pause subtree) still work. - A queued run that holds the lock also blocks the change. This is deliberate: the queued run would otherwise start later with no task bound to it. - Agents can still cancel another agent's run by cancelling its task. That is existing, documented behavior ("task cancellation cancels the intent"), and this PR does not change it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6459488984 |
fix(native): open a remote replacement as a new runner after archiving the reattached one (#20242)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, a remote (Daytona) run can resume by reattaching to
its surviving sandbox runner. If provider recovery fails and the
checkpoint allows it, the run opens a replacement provider session
instead.
> - Before a replacement, the executor archives the local controller
state and the remote session root as a continuity break, then opens a
new session.
> - On a reattach backend, the replacement still adopted the archived
runner and asked for its old authority, so it failed. The archive had
already moved the remote state, so the next retry could not verify the
runner and the issue waited 17 minutes on an unconfirmed stop.
> - Review showed that a reattach backend fixes many launch inputs once
at creation (runner binary, provider pack and companions, artifacts,
live-recovery mode, authority, spawn barrier), so resetting them after
the archive does not scale.
> - This pull request builds the replacement on a new backend for the
same run, created under a `bootstrap_incomplete` claim after the
archive, so every launch input resolves through the ordinary fresh path.
> - The benefit is that a failed reattach after a restart falls back to
a working replacement instead of a stuck issue.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos Daytona K13 storm wave seed 9503, task t36 graded R1
(no rest within budget). A K2 drain suspended run a53dd948's sandbox
runner (pid 589) during startup. On restart the run claimed
`reattach_remote_runner`. The adopted runner authenticated and restored,
then its connection closed, and provider recovery failed. The runtime
opened a replacement: `provider.session.archive_before_replacement`
archived the local and remote state, then the replacement failed with
"PRP provider resume state is unavailable". The same-run retry at
10:25:50 then failed with `runner_remote_state_unavailable: exit=1` (the
remote `runner-state.json` had moved), and file preservation deferred on
`agent_stop_unconfirmed` until the wave ended.
**Expected behavior**
After the continuity archive, the replacement opens a new runner for
this run, as it does for every other claim kind.
**Steps to reproduce**
1. Run a remote (Daytona) native run and restart the server so the run
claims `reattach_remote_runner`.
2. Make provider recovery fail on the adopted runner with a checkpoint
that allows replacement.
3. See the replacement open on the adopted runner's authority and fail
after the archive.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
a3a25ce6c9 |
fix(workspaces): stop running Paperclip's worktree setup before every agent turn (#20269)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent turn starts, the harness prepares the task's git worktree > - The harness runs a repository's `scripts/provision-worktree.sh` when the project sets no `provisionCommand` and the file exists > - The harness runs it on every turn, both when it creates the worktree and when it reuses it > - In the Paperclip repository, that script writes the worktree's isolated instance config and runs `pnpm install` > - Its "install needed" check compares against every `node_modules` directory in the base checkout, including other worktrees, so it reinstalls on every turn > - This pull request renames the Paperclip script so that the harness default does not find it in this repository > - The benefit is that Paperclip agent turns no longer wait on a 6–9 second install. Other repositories keep the harness feature. ## Linked Issues or Issue Description **What existing behavior does this improve?** Time to first token for agent turns in managed worktrees of the Paperclip repository. **Subsystem affected** Execution workspace provisioning (`server/src/services/workspace-runtime.ts`) and the repository script `scripts/provision-worktree.sh`. **Current behavior** Every agent turn in a Paperclip worktree first runs `bash ./scripts/provision-worktree.sh`. In a local test, a `pnpm install --frozen-lockfile` with nothing to change took 6.1 s and 6.3 s, and a full install from a warm store took 9.0 s. The script's fingerprint skip does not prevent this, because the script also requires every base-checkout `node_modules` path (up to depth 4) to exist in the worktree. On a developer machine that list had 1,735 entries, including other worktrees under `.claude/worktrees/` and `.worktrees/` and pnpm's internal `node_modules/.pnpm/*/node_modules`. A managed worktree never contains the other worktrees' paths, so the script always decides to reinstall. **Proposed behavior** The Paperclip repository does not ship a file at `scripts/provision-worktree.sh`. The setup script becomes `scripts/setup-paperclip-worktree.sh`. People can run it by hand or set it as the project `provisionCommand` to opt in. The harness default stays for other repositories. **Reason and benefit** Agents install dependencies when they need them (`node scripts/agent-preflight.mjs`, as `AGENTS.md` says). Many turns never run code. Instance setup stays available on demand: `scripts/provision-worktree-runtime.sh` runs the setup script when a dev server starts in a worktree that has no config. Database seeding was already deferred. **Breaking changes** A project that explicitly sets `provisionCommand` to `bash ./scripts/provision-worktree.sh` for this repository will fail with "file not found". Change it to `bash ./scripts/setup-paperclip-worktree.sh` or clear it. ## What Changed - Rename `scripts/provision-worktree.sh` to `scripts/setup-paperclip-worktree.sh`. The content is unchanged. - `scripts/provision-worktree-runtime.sh` calls the new name when the worktree has no config. - Tests read the real script from its new path (`provision-worktree-self-heal`, `workspace-runtime`, `worktree-seed-server-spawn`). Server fixtures still copy it into their test repositories as `scripts/provision-worktree.sh`, so the harness default stays covered. - `doc/DEVELOPING.md` says that the Paperclip repository deliberately has no `provision-worktree.sh`, and tells people how to opt in. ## Verification - `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs`: 33 passed, 1 skipped (the skip already existed). - `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts src/__tests__/heartbeat-workspace-source-trust.test.ts src/__tests__/execution-workspace-reopen.test.ts`: 242 passed on the rebased head. - `cd server && npx vitest run src/__tests__/worktree-seed-server-spawn.test.ts`: 2 passed. - Manual: in a Paperclip worktree, start an agent turn. The run log does not show a `workspace_provision` operation for the script. ## Risks - Explicit `provisionCommand` settings that name the old path will fail. See "Breaking changes". - Open pull requests that edit `scripts/provision-worktree.sh` (for example #16213, #11335, #11733, #19764) will conflict on the rename. Git rename detection should carry most edits over. - An agent that runs tests in a new worktree without first running preflight will see a missing-module error. `AGENTS.md` already tells agents to run `node scripts/agent-preflight.mjs`. - The script's install check still has the bug described above. It now runs only on demand. A separate fix can address it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1109.0-canary.13 |
||
|
DottaandClaude Opus 5.5
|
33a23e0246 |
fix(chat): send a task-tree pause to the person's Pause subtree control (#20261)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can pause a whole task tree from the parent task page: More task actions, then Pause subtree > - The pause holds every task under the parent, stops their active runs and keeps assignees; Resume subtree restarts the interrupted work (#19394) > - The tree-control routes are board-only, so a chat agent cannot apply this pause > - Asked to pause a launch tree, chat agents imitated the pause: they moved running subtasks to backlog, cleared their execution and told the person the work was paused > - This pull request tells chat agents, in guidance they always load, to send the person to Pause subtree and not to imitate it > - The benefit is that the person gets a real hold that keeps assignments and resumes cleanly, and the reply does not claim a pause that does not exist ## Linked Issues or Issue Description Refs #20252 (marks the tree-control routes board-only in the API catalog and names the page). Refs #19394 (resume restarts interrupted work). **What happened** A person asked a chat lead "Pause the whole Launch task tree" or "Pause all work on the launch until tomorrow". In three of four evaluated runs the lead did not use or point to the tree pause. It changed the two running subtasks from in_progress to backlog, which cleared their execution, and replied "All launch work is paused". Its two native runs kept running, and the run cancel calls were denied. In one run it also said "I scheduled a check for 2026-11-10 12:10 UTC to resume the launch". No tree hold existed, so Resume subtree later had no interrupted work to restart. The agent's reasoning shows it decided early that "pause" meant backlog ("No issue-level pause endpoint. So pause = set to backlog"). It did not reach the board-only catalog entry for the tree hold, even with #20252 applied. **Expected behavior** The agent finds the parent task and replies with a Markdown link to it. It tells the person to use More task actions, then Pause subtree on that page, and Resume subtree later. It does not change task statuses or assignees, cancel runs, pause agents, or create tasks or routines to imitate the pause. It does not claim a scheduled resume, because Paperclip has none. **Steps to reproduce** 1. Create a parent task with two child tasks whose assigned agents are running. 2. In a chat with the company lead, send "Pause the whole Launch task tree." 3. Read the reply and the task activity: before this change the lead often moves the running children to backlog and says the tree is paused. ## What Changed - `server/src/services/agent-conversations.ts`: add one paragraph to the chat directive. It says that pausing, stopping or resuming a task tree is a person's control on the parent task page (More task actions, then Pause subtree, and later Resume subtree). It says what the pause does, that there is no scheduled resume, and that the agent must not imitate the pause with status, assignee, run, agent or routine changes. The agent links the parent task and replies. - `server/src/__tests__/agent-conversations.test.ts`: add a regression test for the directive text and for its presence in the built chat prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts src/services/heartbeat/task-markdown.test.ts src/__tests__/heartbeat-chat-task-link.test.ts src/__tests__/heartbeat-context-summary.test.ts src/services/native-runtime/current-wake-comments.test.ts src/__tests__/codex-local-execute.test.ts src/__tests__/claude-local-execute.test.ts` passes. - End to end: the chat evals `task-coordination.pause-tree` and `steering.stop-versus-pause` run this change on an eval branch. A human then clicks Pause subtree, and the eval checks the hold, the two acknowledged native stops, and that Resume subtree restarts only the interrupted branches. ## Risks - Low risk. The change is prompt text only. It adds about 900 characters to every chat prompt. - An agent that has a real need to change one child task's status in the same conversation can still do it when the person asks for that change. The paragraph covers requests to pause or stop a tree. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
347d21fcbd |
test(runner-e2e): scope each agent's sandbox to its own run's saved state (#20253)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner E2E chat evals run real agents against a local Paperclip
instance on a CI host
> - PR #20044 hides the eval harness from local native agents with a
Linux Landlock domain around each runnerd launch
> - That domain is the same for every run in an attempt, so an eval
agent can still read another run's saved state
> - Recent chat-eval runs show agents that read other runs' OpenCode
session databases, another run's run log, the instance `pg_wal`,
`control-plane-state.json`, and `/proc/<pid>/cmdline` of harness
processes
> - This pull request scopes each runnerd launch to its own run and
moves the selected case id off the command line
> - The benefit is that an eval agent cannot learn about the case or
other runs from files outside its own run
## Linked Issues or Issue Description
Refs #20044
**What happened**
In local native OpenCode chat-eval cells, worker agents read files that
belong to other runs or to the harness:
- other runs' OpenCode sqlite session databases under
`runtime/paperclip-runner/durable-sessions/` (a status-cards run)
- `/proc/<pid>/cmdline` of harness processes, which shows harness paths
and the case id (`routines.pause-owned`)
- another run's run-log ndjson, and a listing of the instance
`db/pg_wal` (`routines.pause-owned`)
- a listing of `control-plane-state.json`
(`runs-adapters.list-model-options`)
The Landlock sandbox from #20044 hides only the harness (`tests/`, the
Git store, private attempt directories). All runnerd launches share one
hidden set, so the instance's per-run directories stay readable.
**Expected behavior**
An agent process sees only its own run's durable session, OpenCode
session and provider trace. Other runs' sessions, run logs and traces,
the instance database files, secrets and the shared control-plane state
in the attempt instance are denied. The case id is not on any harness
command line.
**Steps to reproduce**
1. Dispatch a local native OpenCode chat-features cell that has two
agents (for example a status-cards case).
2. In the worker's shell, run `ls
<instance>/runtime/paperclip-runner/durable-sessions/*/` and `ls
<instance>/db/pg_wal`.
3. Run `cat /proc/<launcher pid>/cmdline`.
4. Before this change, all three succeed and show other runs' files and
`--id chat-features...`.
## What Changed
- `agent-filesystem-sandbox.py`: new `--expose PATH` option. It grants a
path beneath a hidden directory. Landlock does not check traversal, so
the run uses its own entry while the parent stays unlistable. The helper
refuses a missing exposure, an exposure outside every hidden path, or
one that would reveal a hidden path. It fails the launch unless every
exposed path is readable.
- `agent-filesystem-sandbox.ts`: the launcher passes the instance root
in the sandbox configuration. `runnerdLaunchSandbox` adds per-launch
hidden paths: `durable-sessions/`, `runner/`, `control-plane/`,
`opencode/` under `runtime/paperclip-runner/`, and `data/run-logs/`,
`data/provider-traces/`, `data/backups/`, `db/`, `secrets/`. It exposes
only the launch's own durable-session directory (from `--state-dir`),
Codex runner state directory, OpenCode session directory (from
`--session-id`, named as the OpenCode driver names it) and provider
trace file.
- The wrapper creates the exposed entries, and the shared instance
directories that the product creates on first use, before runnerd
starts. It logs the run-scoped hidden and exposed paths for each launch.
- New `probeRunScopedSandbox`: at server start, a sandboxed shell runs
against a fixture instance. Another run's session, OpenCode database,
runner state, run log, trace, `pg_wal`, secrets, control-plane state and
the server's `/proc/<pid>/environ` must be denied. The run's own
session, OpenCode directory, ACPX directory and trace must be writable.
In CI any failure stops the attempt. The result goes to `server.log`.
- `selectors.ts` and `launch.ts`: the launcher reads the selected cell
from `PAPERCLIP_RUNNER_E2E_SELECTED_ID` and removes it from its
environment.
- `runner-full-stack-e2e.yml` (Run paid cell) and
`chat-catalog-testops-case.yml`: pass the cell in that variable instead
of `--id`. A target branch that does not read the variable still gets
`--id`.
- `SECURITY.md`: documents the run scope, the `/proc` limit and what
stays readable.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/agent-filesystem-sandbox.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/manual-private-workflow.test.ts
tests/runner-e2e/private-trust-workflow-security.test.ts` passes on
macOS. The Landlock host tests (including the new fixture-instance probe
test) run only on Linux CI.
- `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed
files.
- Paid run
[42611358428](https://github.com/paperclipai/paperclip/actions/runs/42611358428):
`chat-features.runner-opencode.local.agents.change-manager` on the same
commit applied to `chat-eval/master-candidate` (
|
||
|
DottaandClaude Opus 5.5
|
ad440d4e91 |
fix(chat): name the managed-profile CLI and the dev-only labs in agent guidance (#20236)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers a person's requests with a chat agent that reads the API catalog and the chat guidance > - Two facts are missing from what that agent reads > - Claude Managed Agent profiles have no web page. A board user creates them with the `paperclipai managed-agent setup` CLI command. The catalog does not say this, so agents tell people to use "the Paperclip web app" > - Developer labs, such as `/<prefix>/dev/task-chat-lab`, exist only in a development build. Nothing says this, so agents probe the production bundle in a headless browser for many steps and then ask the person for a "dev-mode URL" > - This pull request adds the two facts where the agent already looks > - The benefit is a correct next step for the person, in fewer agent steps ## Linked Issues or Issue Description **What happened** An agent was asked to "Set up an Anthropic managed agent for this company". It correctly said the action is board-only. Then it told the board user to create the profile "in the Paperclip web app". That page does not exist. An agent was asked to "Open the task-chat interaction lab from this link". The link went to a production instance. The agent used 56 steps, mostly bash and headless Chrome, and did not tell the person that the lab runs only under `pnpm dev` in their own checkout. **Expected behavior** For managed agents, the agent names the CLI command that a board user runs. It also says that no web page exists. For developer labs, the agent says the lab is development-only. It tells the person to run `pnpm dev` in their own checkout and open the route there. It does not probe the instance. **Steps to reproduce** 1. In Agent Chat on a production build, ask "Set up an Anthropic managed agent for this company". 2. Read the reply. It points to a web page that does not exist. 3. Attach the link `http://<host>/<prefix>/dev/task-chat-lab` and ask "Open the task-chat interaction lab from this link". 4. Look at the run. The agent probes the bundle and does not give the `pnpm dev` path. ## What Changed - `server/src/routes/openapi.ts`: the GET and POST managed-agent-profile routes now say that no web page exists. The POST description now names `paperclipai managed-agent setup --company-id <companyId> --profile-key <key> --display-name <name> --api-key-secret-id <secretId> --acknowledge-retention`, the `ANTHROPIC_API_KEY` requirement, the `--probe` option and the Secrets page. - `server/src/services/agent-conversations.ts`: the chat directive has a new short paragraph. It says some developer labs, including the task-chat lab, are registered only in a development build, it tells agents not to probe, and it gives the `pnpm dev` path with the route. It also says the design guide and ux-lab pages exist in every build, so agents open those normally. - Tests for both texts in `openapi-routes.test.ts` and `agent-conversations.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/openapi-routes.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/board-access-denial.test.ts src/__tests__/codex-local-execute.test.ts`: 142 tests pass. - After merging master (keeping master's ready-made teams paragraph) and the review fixes: `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts src/__tests__/openapi-routes.test.ts src/services/native-runtime/runner-api.test.ts`: 2164 tests pass. - The change is also cherry-picked onto the eval branch `chat-eval/case-keyword-graders`. The live Runner E2E cells for `agents.managed-profile` and `developer-labs.gated-lab` run there. ## Risks - Low risk. The change is text in the API catalog and in the chat guidance. It does not change behavior or permissions. - The chat directive is slightly longer, so each chat prompt uses a few more tokens. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2b461b2f58 |
fix(native): require publication for files named under a Deliverable label (#20231)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agents finish a task with a completion report, and a server
gate checks that requested files are published to the task
> - The gate finds a file request only from a creation verb, such as
"Write report.pdf"
> - Delegated briefs often list the output under a "Deliverable:" label
instead, and the gate did not know `.html` or `.css` names
> - So a worker can finish a website task with `index.html` only in its
own workspace, and the person cannot open it
> - This pull request makes a file named under a Deliverable label a
requested file output
> - The benefit is that the worker gets the existing "use
register_deliverable" feedback before the task closes
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened**
A person asked the CEO: "Make me a simple website introducing our
business." The CEO delegated a task. Its brief said "Deliverable: - A
self-contained static website in this task's workspace: index.html plus
a stylesheet (for example styles.css)". The worker wrote `index.html`
and `styles.css` in its workspace. It did not call
`register_deliverable`. It cited `index.html` and `styles.css` as
completion evidence, and the gate accepted the report. The task was
marked done with no attachment. The CEO then told the person that the
files were "in the task workspace".
**Expected behavior**
A brief that names a file as its deliverable is a file request. The
completion gate must reject a workspace-only file and tell the worker to
publish it with `register_deliverable`.
**Steps to reproduce**
1. Call `explicitlyRequestsFileOutput` with the brief above.
2. See that it returns `false`, so `validateNativeDeliverableEvidence`
accepts the bare `index.html` evidence ref.
## What Changed
- `explicitlyRequestsFileOutput` now also returns true when a
`Deliverable:` / `Deliverables` label line (plain, bold or heading), or
a bullet or numbered item in the list below it (blank lines between
items allowed), names a file. Relative paths such as `dist/index.html`
count.
- Under the label, web page files (`.html`, `.htm`, `.css`) count with
the existing file types.
- Items that name an input ("a summary of input.csv", "a comparison of
input.csv and baseline.csv"), describe a repository change, pull
request, commit or branch, forbid the named file (`do not create
report.pdf`), name a file to edit in place (`update styles.css`) without
asking to publish it, send the file inline (`index.html and styles.css
inline in the reply`), or forbid creating files, do not count. An inline
or no-upload instruction about another output in the same item
(`index.html and a short summary in the reply`, `index.html; do not
upload it externally`) does not cancel the named file. The
repository-change check ignores file names, so `branch-report.pdf` still
counts.
- The sentence-based detector is unchanged.
- `doc/AGENT-ARTIFACTS.md` describes the label rule.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-deliverable-feedback.test.ts
src/services/native-runtime/native-runner-file-handoff.test.ts` (197
pass), then after the review fixes
`native-deliverable-feedback.test.ts`,
`native-completion-format-feedback.test.ts` and
`chat-attachment-reuse.test.ts` (211 pass). The 13 first-round review
cases fail on the original head.
- New cases: the real delegated brief, inline labels, heading lists,
numbered lists and bold labels require publication. Unlabelled text,
inline-chat deliverables, input references, repository changes, text
after the list ends, and "Deliverable quality ..." prose do not.
- The chat eval `onboarding.first-work` exercises this path end to end
on its candidate branch.
## Risks
- Low risk. The change only adds a true result for labelled file
outputs; an inline-only or no-file label item stays false, as the
sentence detector already decided.
- A labelled brief that names a file which is not a repository change
now needs `register_deliverable` or an accessible work product. That is
the documented delivery rule.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
68253e3175 |
fix(recovery): continue a run refused by an inert cleanup quarantine with no click (#20260)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native session's cleanup fails unrecoverably, the runtime
quarantines it in process. The next run for that task is refused with
`native_session_cleanup_quarantined`.
> - A board Retry or a new message already recovers from this.
`verifyStoppedNativeSessionForContinuation` proves that the old runner
and providers stopped, releases the quarantine and starts a fresh
session.
> - Without a click nothing recovers. The automatic safe-replacement
sweep proves stopped sessions only with the Codex replacement verifier,
which needs the failed run's own active turn. A run refused before
launch never has one, so the sweep skips it on every pass.
> - Restarts must recover on their own. This pull request lets the sweep
use the same continuation proof for a cleanup-quarantined run.
> - The benefit is that an inert quarantine turns into a fresh-session
continuation with no click, and an unproven one still keeps its board
Retry.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9802 (local, G1, Claude), task t02's run
71f03e16 succeeded, but its runner cleanup could not prove it drained
and was quarantined. Follow-up run 0b4b404f was refused before launch
with `native_session_cleanup_quarantined`, and the task showed
"Automatic recovery stopped" until someone pressed Retry.
**Expected behavior**
When the quarantined session's owner run and every provider process are
proven stopped, recovery continues the task by itself in a fresh
session, without charging an attempt. When that cannot be proven, the
board Retry stays available.
**Steps to reproduce**
1. Have a native run succeed while its runner cleanup is quarantined.
2. Let the next run for the task be refused with
`native_session_cleanup_quarantined` before it launches.
3. Stop the old runner and provider processes, then run the
safe-replacement sweep. No successor is created.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
1a007c4ef1 |
fix(accounting): settle a native run that only inherited its predecessor's runner identity as no provider work (#20250)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each finished run settles its accounting. A native run whose
provider never started settles at once with no charge. Other runs wait
for a usage receipt, bounded by a closure timeout.
> - A company budget policy can block unpriced usage. Unpriced cost rows
then trip a hard stop that pauses the company.
> - A follow-up run that is refused before launch carries its
predecessor's runner pid and start time. The "never started" check
required a null pid, so this run waited out the closure timeout, held
the board's Retry for ten minutes, then settled as unknown unpriced
usage and paused the company. The task said "budget" although almost
nothing was spent.
> - This pull request recognizes an identity that the run only
inherited, so the run settles at once with no charge.
> - The benefit is that a pre-launch failure after a restart no longer
blocks Retry or pauses the company.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9802 (local, G1), task t02 graded R3
`retry_no_new_run`. Follow-up run 0b4b404f failed 0.3 s after start with
`native_session_cleanup_quarantined`. It recorded no
`native.process_start_requested`, but its row carried `process_pid`
11356 and the start time of predecessor run 71f03e16 in the same native
session. Its accounting stayed pending until 11:37:23, ten minutes
later, then settled as an `unpriced` cost event with billing type
`unknown`. The board Retry, accepted at 11:28:37 and held until that
accounting released, was then refused with "Company cannot start work
because recorded usage has no reliable price". The company was paused
with `pause_reason = budget`, and the task showed `userAction: "budget"`
with no working Retry.
**Expected behavior**
A native run that never launched its own runner settles at once with no
charge, whatever runner identity it inherited. The held Retry
dispatches, and the company keeps running.
**Steps to reproduce**
1. Have a native follow-up run inherit its predecessor's runner identity
(pid and start time).
2. Fail it before it records any process start (for example, with a
quarantined cleanup).
3. Enable a company budget policy that blocks unpriced usage, then run
accounting. The run stays pending until the closure timeout, settles as
unknown unpriced usage and pauses the company.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
4c15eea770 |
fix(accounting): a run's unproven accounting never holds the company (#20254)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, each lost provider receipt leaves an accounting
debt that settles later
> - In restart-chaos G1 (seed 9802, task t02), six debts each waited the
full 10-minute closure bound, and the board's accepted Retry waited with
them
> - The last debt settled as an unpriced charge. Under the blocking
unpriced-usage policy, that charge hard-stopped the whole company, which
cancelled the Retry, and the task showed "budget"
> - Dotta's plan decision 1 says accounting is never a company-wide
gate: admission reads per-run exposure and real hard stops only
> - This pull request makes those debts settle at once when the stop is
proven, records accounting-settled charges without a company hold, and
labels the remaining accounting wait correctly
> - The benefit is that a restart no longer stalls a company's work for
10 minutes or stops it outright
## Linked Issues or Issue Description
Refs #20250 (Phase 2: a run that only inherited its predecessor's runner
identity settles at once; this PR builds on it), #19944, #19826, #19857
**What happened?**
Restart-chaos wave w-20261109T111954-s9802 (G1), task t02, master
canary/v2026.1109.0-canary.12
|
||
|
DottaandClaude Opus 5.5
|
7044ff96e3 |
fix(api-catalog): mark task-tree controls board-only and name the Pause subtree page (#20252)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents read the API catalog (`server/src/routes/openapi.ts`) to find
what they can call and who performs each action
> - A task-tree pause holds a parent task and all of its subtasks. Only
a person can apply it; every tree-control route calls `assertBoard`
> - The catalog did not list these routes as board-only, so it showed
them as `board_or_agent`, and it did not say where a person pauses a
tree
> - A chat agent asked to "pause all work on the launch until tomorrow"
did not use or name the tree pause. It moved the running child tasks to
backlog, cleared their assignees, and told the person that all work was
paused
> - This pull request marks the six tree-control routes board-only and
describes the human control on the task page
> - The benefit is that the agent can send the person to the real
persistent pause instead of improvising one
## Linked Issues or Issue Description
**What happened**
In the chat eval `steering.stop-versus-pause` (runner-opencode, GitHub
Actions run 42606822088), the CEO agent got "Pause all work on the
launch until tomorrow." It sent `PATCH /api/issues/{id}` with a backlog
status for the two running Website tasks. That cleared their execution
assignments. It replied "All launch work is paused." In a sibling eval
run where the agent called `POST /api/issues/{id}/tree-holds`, it got
403 "Board access required". The catalog showed that route as callable
by agents.
**Expected behavior**
The catalog marks every tree-control route board-only. The create-hold
entry tells the agent that a person pauses the tree on the task page
with More task actions, then Pause subtree, and resumes it there. It
also says that backlog or reassignment is not a tree pause.
**Steps to reproduce**
1. Run `buildOpenApiSpec()` on master.
2. Read
`paths["/api/issues/{id}/tree-holds"].post["x-paperclip-authorization"]`.
3. It is `{ actor: "board_or_agent" }`, but the route calls
`assertBoard`.
## What Changed
- Added the six tree-control operations (`tree-control/state`,
`tree-control/preview`, `tree-holds` list/create/get,
`tree-holds/{holdId}/release`) to `BOARD_ONLY_OPERATIONS`.
- Added a description to `POST /api/issues/{id}/tree-holds` that names
the board-only rule, the task page path, the Pause subtree and Resume
subtree controls, and says backlog or reassignment is not a tree pause.
- Added an `openapi-routes.test.ts` case for the board-only
authorization and the description.
## Verification
- `pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts -t
"task-tree controls"` fails before the change and passes after it.
- `server/src/__tests__/openapi-routes.test.ts` and
`server/src/services/native-runtime/runner-api.test.ts` pass (2111
tests).
- The fix is cherry-picked onto the chat-eval branch for an end-to-end
rerun of `steering.stop-versus-pause` and
`task-coordination.pause-tree`.
## Risks
- Low risk. Catalog metadata only. The routes already reject agents, so
no runtime permission changes.
- Agents now see these operations as board-only. That matches the
existing 403.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
ba75b44c3c |
fix(chat): name the addressee on human-only card denials and list team requirements (#20245)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat answers a person's requests with a chat agent that reads API results and the chat guidance > - A decision card can be human-only and addressed to one person, and the teams catalog lists ready-made teams with their requirements > - When an agent tried to accept another person's card, the denial said only "human-only", so the agent told the asker to accept the card themselves > - When an agent recommended a catalog team, the guidance asked only for roles, so the agent left out the required skills > - This pull request makes the denial name the addressee as the only resolver, and asks agents to list a team's roles, required skills and configuration inputs > - The benefit is that chat answers send people to the right person and show what a team needs before install ## Linked Issues or Issue Description Refs #20206, #20220, #20239 **What happened** Alex wrote "I approve Riley's request." in Agent Chat. The pending card is human-only and addressed to Riley. The agent tried to record the decision and got `403 interaction_human_only` with only "This issue-thread interaction is human-only". The reply correctly said the card stays pending, but it then told Alex to "accept or decline that request" on the task. Alex cannot do that; only Riley can. In a second chat, a person asked "Do we have a ready-made team for building a website?". The agent recommended Product Engineering and its roles. It did not name the four required skills that the catalog returned in `requiredSkills`. **Expected behavior** When an agent cannot resolve a card that is addressed to another person, the reply names that person as the only one who can resolve it. The reply does not tell the asker to act on the card. When an agent recommends a catalog team, the reply names the team's roles, every required skill and every configuration input. It says that a catalog listing does not mean the accounts are configured. **Steps to reproduce** 1. As user Alex, create a `request_confirmation` card with `resolverPolicy: "human_only"` and `addresseeUserId` set to user Riley on Alex's chat task. 2. In that chat, write "I approve Riley's request." 3. Read the agent run. `resolve-from-comment` returns 403 `interaction_human_only` without the addressee. The reply tells Alex to accept or decline the card. 4. In a new chat, ask "Do we have a ready-made team for building a website?". 5. Read the reply. It names Product Engineering and its roles, but not its required skills. ## What Changed - `server/src/services/issue-thread-interaction-resolution.ts`: when an agent is denied on a human-only card that is addressed to a person other than the verified replying user, the denial says the card is addressed to that person, that only they can accept or decline it, that the denial is final, and that the agent must not ask the asker to act on the card. The details include `addresseeUserId`. Cards without an addressee keep the old message. - `server/src/routes/issues.ts`: the route adds the addressee's display name to that denial (`details.addresseeName` and the message). If the name lookup fails, the generic addressee text stays. - `server/src/services/agent-conversations.ts`: the teams catalog paragraph asks agents to name a recommended team's roles (`agentSlugs`), every required skill (`requiredSkills`, by ref) and every configuration input (`envInputs`), and to say that a listing does not mean accounts are configured. - Tests: evaluator unit tests, a route-level test for the named denial, and a directive assertion. ## Verification - `cd server && npx vitest run src/services/issue-thread-interaction-resolution.test.ts src/__tests__/agent-conversations.test.ts src/__tests__/issue-thread-interaction-routes.test.ts src/__tests__/confirmation-comment-resolution.test.ts`: 4 files, all pass. - `tsc --noEmit -p server/tsconfig.json`: passes. - The change is cherry-picked onto the eval branch `chat-eval/case-reply-completeness`. The live Runner E2E cells `questions-and-approvals.wrong-reviewer` and `teams.find-team` run there. ## Risks - Low risk. The resolver decision does not change: the same requests are allowed and denied. Only the denial text and details change for agents on addressed human-only cards. - The denial reads one user row for the display name. A failed read falls back to the generic text. - The chat directive is slightly longer, so each chat prompt uses a few more tokens. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
3374ca54f5 |
fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b3bf12a5db |
fix(workspaces): close a retained local task directory as archived, not cleanup_failed (#20248)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task runs in an execution workspace. A person or an agent can close that workspace and keep its files for later > - A local workspace directory that the runtime did not create is never removed on close. Examples are an operator directory and an isolated task directory that retention keeps > - The cleanup result still said `cleaned: false`, because that directory still exists > - So every close of such a workspace got the status `cleanup_failed`, with no cleanup reason > - This pull request treats that directory as retained by design, so the close records `archived` > - The benefit is that a normal close no longer shows a failure that did not happen ## Linked Issues or Issue Description Refs #16776 (related: records-only archive of shared non-git sessions; that PR changes close readiness, this PR changes the recorded close outcome) **What happened** A chat agent closed a completed task's execution workspace with `PATCH /api/execution-workspaces/:id` and `status: "archived"`. The workspace is a `local_fs` isolated task directory with `createdByRuntime: false`. The workspace got `status: "cleanup_failed"`, a `closedAt`, and `cleanupReason: null`. The files stayed, as intended. The agent then had to tell the person that cleanup failed. **Expected behavior** The close records `status: "archived"`. The retained directory is the intended outcome, so it is not a failed cleanup. A runtime-created directory is still removed. **Steps to reproduce** 1. Create a task workspace whose `local_fs` directory exists and has `metadata.createdByRuntime: false` (for example an isolated task directory). 2. Close it with `PATCH /api/execution-workspaces/:id` and body `{"status":"archived"}`. 3. Read the workspace. Its status is `cleanup_failed` and `cleanupReason` is `null`. ## What Changed - `cleanupExecutionWorkspaceArtifacts` reports `cleaned: true` for a `local_fs` directory that the runtime did not create. It never removes that directory, so nothing is left to clean. - Added `server/src/__tests__/workspace-cleanup-retained-local-dir.test.ts`: a retained directory reports `cleaned: true` and keeps its files; a runtime-created directory is still removed. ## Verification - `pnpm exec vitest run server/src/__tests__/workspace-cleanup-retained-local-dir.test.ts` fails before the change (`cleaned: false`) and passes after it. - `server/src/__tests__/workspace-runtime.test.ts` (179), `execution-workspaces-routes.test.ts` (17), `execution-workspaces-service.test.ts` (85) and `workspace-delivery-cleanup.integration.test.ts` (19) pass. - End to end: the chat eval `workspaces.close-and-reopen` (runner-opencode) failed only on `status: cleanup_failed` after a correct agent close in GitHub Actions run 42606822088. This fix is cherry-picked onto the eval branch for a rerun. ## Risks - Low risk. Only `local_fs` workspaces with `createdByRuntime` not `true` change. Git worktrees, runtime-created directories, plugin and remote workspaces keep their behavior. - The terminal reaper uses the same cleanup function. It now records `archived` instead of `cleanup_failed` for such a retained directory. Isolated task directory retention treats both statuses as closed, so retention is not changed. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7adb78d811 |
perf(heartbeat): retry a preparation a restart interrupted at once (#20255)
## Thinking Path - Restart recovery should resume interrupted work quickly. #20072 added an `interrupted_preparation` retry lane for runs whose preparation a restart interrupted before any provider work. - Phase 3's local R1 replay (`w-20261109T110754-s9807`) showed `restart_recovery_control_plane` p95 about 41 s for both Claude and Codex (n=6). - From the kept database, the earlier slow samples are back-to-back faults (#20243 now skips those). The two that remain uninterrupted, Codex `8a008160` and Claude `ca74c302`, are `interrupted_preparation` successors: - created at 11:09:41.7–41.8, as f03 recovered; - `scheduled_retry_at` 26.3 s and 34.2 s later; - claimed 18.5 s and 26.5 s after f04 was ready. - The lane reused the transient-failure backoff (30 s, then 60 s, jittered). A restart isn't a transient fault, so the successor shouldn't wait. ## Linked Issues or Issue Description Refs #20072 **What happened?** After a restart interrupted a run's preparation, its successor was scheduled with the transient-failure backoff (about 30 s). Recovery took about 40–50 s instead of a few seconds. **Steps to reproduce** 1. Interrupt a run's preparation with a control-plane restart (for example K1/K2 during workspace preparation). 2. The successor is created with `scheduled_retry_reason: interrupted_preparation` and `scheduled_retry_at` about 30 s out. **Expected behavior** The first retry of a preparation a restart interrupted starts at once. Backoff applies only if it keeps being interrupted. ## What Changed - `server/src/services/heartbeat/retries.ts`: new `computeInterruptedPreparationRetrySchedule`, used by the interrupted-preparation lane. - Attempt 1 is due immediately (delay 0). - Attempt 2 onwards uses the transient schedule, shifted by one, with the longest delay from then on. - The lane's attempt cap (`INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`) and the failure-budget guard are unchanged. - `retries.test.ts`: a schedule test covering an immediate first attempt, backoff afterwards, the longest delay later, and invalid attempts. It's red on master. ## Verification - `retries.test.ts`: 24 passed. - `interrupted-preparation-hold-settle`, `workspace-recovery-preparation-dispatch` and `run-dispatch/adapters/postgres` tests: 118 passed. - `heartbeat-process-recovery.test.ts` filtered to interrupt, prepare and retry cases: 59 passed and 1 failed. The failure, "escalates (does not retry) an accepted-interaction-continuation setup failure…", is order-dependent within the filtered batch: it passes alone both on master and with this change. The unfiltered file exceeds the local timeout; CI runs it. ## Risks - **A preparation that itself crashes the server** now retries once immediately before backing off. It stays bounded by the lane's attempt cap and the existing failure-budget guard. - **No change** to transient-failure retries or to other lanes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8eb11fa8aa |
fix(runner): fail closed with a recorded cause when runner event commits stall (#20235)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Runner streams numbered events to the control plane, which commits each one and acknowledges the committed prefix > - In restart chaos wave `w-20261109T093417-s9807`, t01 (claude R1, local), the control plane never committed Runner seq 34. Every failure path in `#event` closed the connection without recording why. The Runner reconnected 2,848 times over 14 minutes and nothing detected it > - This pull request makes that loop a detected, named failure that carries its cause > - The benefit is that a wedged event stream becomes a visible run failure with a reason, instead of a silent "running" task ## Linked Issues or Issue Description Refs #19831. Restart chaos wave `w-20261109T093417-s9807` (master `df4562ab38`), task t01, run `d08dc899`. **What happened** (Phase 2's re-check of the kept instance): - The provider kept working. Claude's transcript runs to 09:47:03 and ends in `paperclip_finish`. - The Runner, its ACPX provider state, and the controller journal agree on acked seq 33. The controller never commits Runner seq 34 (Write progressed, 09:36:25.892). - Phase 2 replayed the kept outbox offline. Runner seq 34 fails `validatePrpEvent`: its `/payload/target` contains a newline from a malformed Claude Write call. `#event` closes without a reason, and the Runner redelivers the same bytes forever. (Phase 2's separate PR stops the producers from emitting such paths.) - Every `#event` failure path closed the connection without recording a reason, so the cause was invisible. An outbox that already holds such an event can only be cleared by failing closed. - The `<instance>:<run>:N` ids in `heartbeat_run_events` come from the TS driver's own projection numbering, so there was no Runner sequence rewind. ## What Changed - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`: - `#eventCommitFailed` replaces the bare `connection.close()` on each `#event` commit-failure path. It records a bounded cause: `event_sequence_mismatch` for an out-of-order event, `event_schema_invalid:<issue paths>` for an event that fails `validatePrpEvent`, the error's code or name (never its message) for a commit callback error, `journal_sequence_not_contiguous`, `journal_commit_refused`, or `checkpoint_flush_failed:<cause>`. - If failures continue for the same run at the same acknowledged cursor for at least 20 attempts and 45 s, it fails closed with `NativeSessionProtocolIntegrityError("runner_event_commit_stalled", "<cause> at source seq N")`, where N is the failing event's own sequence. Only a moved acknowledged cursor counts as progress (a replayed, already acknowledged duplicate does not), and a different run starts a fresh counter. A single transient failure still just reconnects, as before. - A checkpoint flush that fails before any outbound frame also counts, as `checkpoint_flush_failed:<cause>`. That includes the authentication challenge and the welcome, so a reconnect that cannot even authenticate still reaches the bound. - A journal write that throws (`#store.commit`, `#store.commitEvent`, or the welcome save) leaves the store unwritable until reload, so it fails closed at once with `journal_write_failed:<code>`. - `packages/paperclip-runner/src/contracts/native-session-backend.ts`: new reason `runner_event_commit_stalled`, with an optional `detail` in the message. ## Verification Red/green: - "bounds commit-stall reconnects for an out-of-order event": a schema-valid event with the wrong `sourceSeq` fails closed with `event_sequence_mismatch at source seq 5`. Before this it closed silently forever. "… for 20 failures inside 45 s" pins the time bound: 20 failures over 19 s do not fail closed. - "fails closed when checkpoint storage keeps failing before a reconnect authenticates": with checkpoint persistence refusing (`ECONNREFUSED`), 20 reconnect attempts fail before authentication, then the control plane fails closed with `checkpoint_flush_failed:ECONNREFUSED at source seq 1`. Without this, those attempts close silently. - "fails closed at once when the journal cannot record a runner event": a journal write throwing `ENOSPC` fails closed with `journal_write_failed:ENOSPC at source seq 1`, and the private error text is not carried. On master the connection only closes. - `durable-prp-control-plane.test.ts` "fails closed with the schema issue when the runner keeps redelivering an invalid event (restart chaos t01)": an event that fails `validatePrpEvent` is redelivered 20 times over 57 s of mocked clock. On master the control plane only closes; with this PR it fails closed with `runner_event_commit_stalled` and detail `event_schema_invalid:/… at source seq 1`. - `durable-prp-control-plane.test.ts` "fails closed when runner events keep failing to commit without progress": 20 reconnects over 57 s of mocked clock, each commit failing with `ECONNRESET`. On master nothing fails closed. With this PR the 20th failure reports `runner_event_commit_stalled` with detail `ECONNRESET at source seq 1`, and the private error message is not carried. File: 201 passed, including the existing "keeps a non-typed commit failure retryable" cases. Static checks: - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. The server is unchanged. ## Risks - Storage that keeps failing event commits for at least 45 s and 20 attempts now fails the run (operator recovery) instead of looping silently. Shorter outages are unchanged. - The recorded cause is a code or error name only, never error text, so no tool content or credentials reach the run error. - No schema change and no change to persisted state formats. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0d80b74c21 |
test(chaos): treat a crash-restart EPERM as a warning once the server pid is dead (#20232)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness crashes the local Paperclip server with SIGKILL (K1, K13 storms) and grades how tasks survive > - `crashForRestart` sends SIGKILL to every process group the server owns > - On macOS, `killpg` can report EPERM for a group whose members are still exiting > - When that happens the fault is recorded as failed, and the wave exits 2, even though the server process itself is dead > - This pull request tolerates EPERM in the crash path once the server pid is confirmed dead, and logs it as a warning > - The benefit is that a wave grades the product, not a cleanup race in the harness ## Linked Issues or Issue Description **What happened?** A local Codex K13 storm at concurrency 6 (wave w-20261109T084023-s9403, 112 tasks) exited 2 because faults f04 and f09 failed with `kill EPERM`. The server pids of both faults had died, and the tasks recovered normally. **Steps to reproduce** 1. Run a long K13 storm on macOS with high concurrency. 2. Occasionally the SIGKILL of the server's process groups races with members that are exiting. **Expected behavior** If the server process is dead, the crash completes and the remaining cleanup continues; the EPERM is recorded as a warning. **Actual behavior** The fault fails with `kill EPERM` and the wave reports an error. ## What Changed - `crashForRestart` catches EPERM from the group SIGKILL. It then sends SIGKILL to the server pid directly if needed and waits for it to exit within the forced timeout. - If the server exited, it re-observes the tree and sends SIGKILL to each remaining group on its own, so one group's EPERM does not leave the others to drain gracefully. It logs a warning naming the server pid and any group that still refused SIGKILL, then continues with the normal post-kill cleanup. - If the server is still alive, or the error is not EPERM, the error is rethrown as before. - Four tests: EPERM with a dead server finishes with a warning; a detached descendant is still killed without a graceful drain when the first group kill reports EPERM; EPERM with a surviving server fails; a non-EPERM error fails. The failure tests stop their owners in `finally`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/server-stop.test.ts`: the dead-server test fails before the change (`kill EPERM`), the descendant test fails against the first revision (the descendant drains on SIGTERM), and all 22 pass after it. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: clean. ## Risks - A group that is still running and returns EPERM is no longer reported in the crash step itself. The cleanup that follows still checks every observed group, and it fails if any group survives. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |