mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
f2e449f8f69c7ae68ea12c034bd4823cf93173ae
2830
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f2e449f8f6 |
fix(retry): continue an accepted Retry past chat input held because it predates the stop (#21129)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person's Retry on a connected task (for example a GitHub pull request review) must start work. #21080 made a Retry whose chat replay is refused after the click continue from the recovered saved work > - After #21080 was deployed, accepted Retries on a busy GitHub review agent still did not start > - Those tasks had a saved chat request that arrived while the failed run was still running. Admission holds such a request as `message_predates_stop`, so it can never start alone > - #21080 refused to continue while any saved chat input existed, so this held input blocked the continuation. For a second group, the latest run was an admission that the recovery hold cancelled before dispatch, and the recovered saved work belongs to the earlier failed run > - This pull request lets the continuation adopt input that was held because it predates the stop, and lets the recovered run that held a cancelled admission count as the saved work > - The benefit is that an accepted Retry starts work on these tasks too ## Linked Issues or Issue Description Follow-up to #21080. No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** After #21080 was deployed, `POST /api/issues/:id/retry` returned 202 for blocked tasks of a GitHub review agent (`paperclip_runner`), but no run started. Each Retry receipt changed to `failed` about 5 seconds later (on the 14:48 Retries, 5 seconds for one task and about 75 seconds for two others), with no run and no `issue.retry_continued_from_saved_work` activity. Every affected task had a saved chat request (a newer GitHub review request) with an `issue.execution_waiting` record of reason `message_predates_stop`. The request arrived while the failed run was running, so admission holds it permanently. The interruption projection kept pointing at that held request. In a second group of tasks, the latest run was cancelled before dispatch with `execution_reconciliation_required`, and the recovered saved work belonged to the earlier failed run that held it. Root cause: in `continueRefusedTaskChatRetry` (`server/src/services/task-retry.ts`), any saved chat input from `selectTaskRetryInput` stopped the continuation, so the receipt settled `failed`. The held input can never start by itself, so nothing ran. The recovered-work check also looked only at the latest run, so it missed the recovered run that held a cancelled admission. **Expected behavior** An accepted Retry starts work. Saved input that admission holds because it predates the stop joins the continuation, and the task continues from the recovered saved work. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Let a native run fail with `native_continuation_requires_reconciliation`, and recover its saved work. 2. Save a chat request while that run is still running, so admission holds it with `message_predates_stop`. 3. Click Retry while the chat identity lock is busy, so the source check is deferred and then refused. 4. The route returns 202. The Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** |
||
|
|
4e932fd178 |
perf(attention): index the agents and review paths once for every review of a request (#21132)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task in review needs a path to its next action: a reviewer, a pending interaction or approval, a monitor, a run, a wake, an external reply or a recovery. A review without a path is stalled, and the attention feed tells a person about it. > - The server decides this for each task in review (`listIssueReviewAttentionMap` in `server/src/services/issues.ts`, with the classification in `server/src/services/recovery/issue-graph-liveness.ts`). The task list, the blocked inbox and the attention feed call it. `GET /attention/summary`, which the board sidebar reads on most pages, runs it for every task in review of the company. > - The classification did the company-wide work again for each task. Each check of an agent copied the agent list, indexed it again and walked the reporting chain. Each task scanned every run, wake, interaction, approval, external wait and recovery of the request. Each stalled review also filtered and sorted every agent again, with one such check for each agent. > - Thus the work grew with reviews × agents for covered reviews and with stalled reviews × agents² for stalled reviews. It also allocated a new list and a new index for each check, and the garbage collector of the server paid for that. > - On a hosted instance, an allocation sample of the server put 4.7% of all allocated bytes under this function, and garbage collection used 25% of the main thread. The two slowest board requests there are the attention summary and the task list, which both call it. > - This pull request builds one index for one classification: the agents by id, the eligibility of each agent, the invokable agents of each company in id order, and the paths of each task. Each task then costs its own paths. > - The benefit is that the review classification of a summary request falls from 62 ms to 5 ms for 900 reviews and 60 agents, and from 123 ms to 8 ms for 300 reviews and 180 agents. The request allocates 57% to 91% fewer bytes. Every finding, every review path and every count is the same. ## Linked Issues or Issue Description Refs #20851 and #20858 (they batched the review-wait provenance check and the task read decision of the same requests; this change is the next cost on that path). Refs #19194 and #13729 (open; they change the failed-run lookup of the attention feed, which is a different statement and a different mechanism; this pull request does not touch it). **What existing behavior does this improve?** The time and the memory that the server uses to decide the review attention of the tasks in review, for `GET /api/companies/:companyId/attention/summary`, `GET /api/companies/:companyId/attention`, `GET /api/companies/:companyId/issues` and the blocked inbox. **Subsystem affected** Server: task-graph liveness classification (`server/src/services/recovery/issue-graph-liveness.ts`) and the review-attention map in `server/src/services/issues.ts`. **Current behavior** - `classifyIssueReviewPaths(input, issue)` built `new Map(input.agents…)` for each task and scanned six path lists of the request for each task. - `isInvokableAgent` copied the agent list for each check. `getAgentOrgChainHealth` then indexed that copy again. - `ownerCandidatesForRecoveryIssue` ran for each stalled review. It filtered and sorted all agents, and it checked each agent again in two more loops. Each check did the copy and the index above. - `hasActiveExecutionPath` built a joined list of the runs and the wakes for each blocker it looked at. - `listIssueReviewAttentionMap` searched the review list for each open recovery task (`reviewIds.includes`) and for each review (`issues.find`). Evidence from a hosted instance (sanitized): - Server main thread at 93% of one core. Garbage collection: 25% of the main thread (387 pauses in 32 s, the longest 224 ms). - Allocation sample, 30 s: 77.7 MB/s in total. `listIssueReviewAttentionMap` and its callees: 4.7% of all bytes (`classifyIssueReviewPaths` 4.4%, `classifyIssueGraphLiveness` 2.3%, `reviewFinding` 2.3%). - `GET /attention/summary` took 16.3 s of server time and `GET /issues` up to 15.3 s. Most of that time is waiting behind other work on the same thread (see "Not verified"). **Proposed behavior** One classification indexes its input one time. The cost of one task is the number of its own paths. The cost of the agents is one eligibility decision for each agent. The results do not change. **Reason and benefit** The summary request runs on most board pages. Its CPU time and its allocation also delay every other request, because the server has one main thread and one garbage collector. **Breaking changes** None. `classifyIssueGraphLiveness` and `classifyIssueReviewPaths` keep their signatures and results. `packages/shared` does not change. ## What Changed - `server/src/services/recovery/issue-graph-liveness.ts` - New internal index for one input (`createIssueGraphLivenessIndex`): agents by id, a memo of `getAgentWorkEligibility` for each agent record, the invokable agents of each company in id order, and the paths of each task for the six path lists. The path lists are indexed on first use and keep the order of the input. The company check of each path stays. - `classifyIssueGraphLiveness` builds the index one time and uses it for the review paths, the waiting paths of blockers, the owner candidates and the assignee eligibility of blockers. - New export `createIssueReviewPathClassifier(input)`. It returns the review paths of many tasks from one index. `classifyIssueReviewPaths(input, issue)` stays for single tasks. - The memo uses the agent record as its key, not the agent id. Thus an input that lists one id two times gets the same answers as before. - The eligibility decision itself is still `getAgentWorkEligibility` of `@paperclipai/shared`, with the same agent list. The index calls it one time for each agent, not one time for each check. - `server/src/services/issues.ts` (`listIssueReviewAttentionMap` only): one classifier for all reviews of the request, a set for the review ids, and the review by position instead of a search. No query changes. - Tests and the reproduction (see Verification): - `server/src/__tests__/helpers/issue-graph-liveness-reference.ts`: the classification as it was before this change. It is the oracle of the equivalence tests. - `server/src/__tests__/issue-graph-liveness-equivalence.test.ts`, `server/src/__tests__/review-attention-classification-equivalence.test.ts`, `server/src/__tests__/helpers/review-attention-fixture.ts`. - `server/scripts/benchmark-review-attention.ts`: the reproduction through the real routes. ## Verification Reproduction and before/after measurement. The script starts a local PostgreSQL, seeds a synthetic company, mounts the real `attentionRoutes` and `issueRoutes` with a board user, and sends the two requests over HTTP. With `--baseline-ref` it runs the pre-fix modules in a second process against the same database. It fails unless both versions return the same summary, the same full attention feed, the same task list and the same review attention for every task in review. ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000,9000 --agents=60 pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000 --agents=180 ``` Fixture: one company with tasks of every status (10% in review, 10% blocked), four of five tasks are children, a reporting tree of agents with paused, terminated and pending agents. The reviews cover every kind of path and three ways to stall (31% stalled). The blocked tasks wait behind chains, cycles, cancelled, done, backlog and unassigned blockers. Open recovery tasks (5% of the tasks) name reviews, blocked tasks and done tasks. 6% of the reviews wait for a GitHub reply. A second company with the same shapes is in the same database. Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding. Each version ran in a new process, three rounds in alternating order, 2 warm-up requests and 5 measured requests in each round. A CPU profile of 3 more requests gives the last two columns. | Tasks / agents / reviews | Request | Version | Statements | Server ms, median (min) | CPU ms, median (min) | Main thread busy ms, median (min) | Allocated MB | Review classification ms | GC ms | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 3000 / 60 / 300 (94 stalled) | attention/summary | before | 69 | 121.4 (112.7) | 131.5 (114.4) | 80.2 (73.2) | 170.4 | 16.8 | 5.0 | | 3000 / 60 / 300 (94 stalled) | attention/summary | after | 69 | 105.9 (94.7) | 113.3 (96.3) | 67.3 (58.6) | 73.3 | 2.4 | 3.4 | | 9000 / 60 / 900 (283 stalled) | attention/summary | before | 85 | 280.9 (263.9) | 281.4 (253.4) | 203.7 (186.8) | 509.8 | 62.1 | 30.0 | | 9000 / 60 / 900 (283 stalled) | attention/summary | after | 85 | 224.8 (213.4) | 220.1 (205.8) | 149.9 (142.3) | 219.1 | 4.8 | 6.6 | | 3000 / 180 / 300 (94 stalled) | attention/summary | before | 69 | 241.2 (203.9) | 248.7 (227.4) | 197.1 (171.1) | 950.2 | 123.2 | 21.2 | | 3000 / 180 / 300 (94 stalled) | attention/summary | after | 69 | 127.8 (108.5) | 128.5 (102.0) | 76.7 (64.8) | 87.3 | 7.8 | 5.1 | | 3000 / 60 / 300 | task list (500 rows, 48 reviews) | before | 59 | 102.7 (96.7) | 92.4 (83.6) | 54.7 (48.9) | 73.2 | 2.6 | 2.4 | | 3000 / 60 / 300 | task list (500 rows, 48 reviews) | after | 59 | 106.2 (91.2) | 89.9 (78.8) | 55.2 (44.8) | 63.4 | 0.9 | 1.6 | | 9000 / 60 / 900 | task list (500 rows, 50 reviews) | before | 42 | 194.3 (186.0) | 77.5 (69.1) | 43.3 (38.4) | 57.3 | 1.0 | 1.8 | | 9000 / 60 / 900 | task list (500 rows, 50 reviews) | after | 42 | 189.6 (182.3) | 72.1 (69.5) | 41.9 (39.4) | 57.8 | 1.2 | 2.5 | | 3000 / 180 / 300 | task list (500 rows, 48 reviews) | before | 59 | 124.6 (95.7) | 110.2 (99.4) | 67.8 (58.1) | 165.4 | 15.4 | 3.8 | | 3000 / 180 / 300 | task list (500 rows, 48 reviews) | after | 59 | 104.1 (91.9) | 90.7 (79.9) | 56.2 (47.1) | 78.2 | 2.9 | 5.3 | How to read the table: - "Review classification ms" is the time in `listIssueReviewAttentionMap` and everything it calls on the main thread, for one request, from the CPU profile. It is the direct measure of this change. Before the fix it was the largest block of application code in the summary handler: 22% of the busy time at 3000 tasks, 20% at 9000 tasks, and 61% with 180 agents. `getAgentOrgChainHealth` under `isInvokableAgent` under `ownerCandidatesForRecoveryIssue` had most of it. After the fix it is 4% to 9%. - The scaling: three times the tasks gave 3.7 times the classification time before and 2.0 times after. Three times the agents gave 7.3 times before and 3.3 times after. - "Allocated MB" is all bytes that the server process allocated for one request (`total_allocated_bytes`), including the HTTP client of the script. - Other agents used the same machine during the measurement (load average 8 to 16 on 18 CPUs), so the medians of server time have noise. The profile columns and the allocation column are the stable ones. - The task list changes little with 60 agents, because one page has about 50 reviews. With 180 agents its classification falls from 15 ms to 3 ms and its allocation by half. - Statement counts do not change: this change removes no statement. Red/green regression test for the mechanism (`server/src/__tests__/issue-graph-liveness-equivalence.test.ts`): - `reads each agent and each path a number of times that does not grow with the reviews`. The test counts the reads of the agent status and of the path task id for 20 and for 400 stalled reviews with 40 agents and 360 paths. On the code before this change: `expected { agentStatus: 200800, pathIssueId: 144000 } to deeply equal { agentStatus: 10040, pathIssueId: 7200 }`. After: 316 agent reads and 360 path reads for both sizes (one read of each path). The test also runs the oracle and asserts that its reads grow, so it shows what it catches. - `classifies the review paths of a page from one index`. The same count for `createIssueReviewPathClassifier`. Equivalence tests (old code and new code on the same input): - `returns the findings and review paths of the pre-fix classification for random task graphs`. 400 seeded random graphs of two companies. Each graph has reviews with and without paths, blocked chains and cycles, terminal blockers, relations and paths that name the other company or a missing task, reporting chains with cycles, missing managers and managers of the other company, and ids that the input lists two times. The test compares the findings of the graph and the review paths of every task, for both `classifyIssueReviewPaths` and the new classifier. It asserts that the graphs reach all six finding states and all nine kinds of path, each at least 20 times. - `returns the pre-fix findings for blocked chains, cycles and terminal blockers`. One named graph with the expected state of each finding, so the comparison cannot pass on two wrong answers. - `server/src/__tests__/review-attention-classification-equivalence.test.ts`, on real PostgreSQL through the real routes. The test wraps the classification module: every classification that a request runs is also run by the oracle on the same input, and a difference fails the test. - `gives every review the paths the fixture was built with, and the pre-fix classification of every request`. `GET /issues?status=in_review`, `GET /issues` and `GET /issues?attention=blocked` (the blocked inbox, whose graph has the chains, cycles and terminal blockers). Each of the 64 reviews has the path kinds and the stalled reason that the fixture states for it, including reviews that wait for an external reply and reviews that open recovery tasks cover. - `counts the reviews that need a person, leaves out the ones a person dismissed, and reads one company`. `GET /attention/summary` and `GET /attention`: the review count and the review cards are the reviews that the fixture built to need a person. The second company has its own counts and none of the first company's tasks. Three dismissed cards leave the count and stay stalled in the task list. - The benchmark above compares the complete responses of both versions through the routes at each size (summary, full feed, task list, review attention of every review). Commands run locally: - `pnpm --filter @paperclipai/server exec vitest run` for `issue-graph-liveness-equivalence`, `review-attention-classification-equivalence`, `issue-liveness`, `recovery-classifiers`, `issue-review-attention`, `issue-blocker-attention`, `blocked-inbox-review-wait-scope`, `heartbeat-issue-liveness-escalation`, `attention-retention-work` and `attention-service`: 210 tests passed. In that run the `afterAll` hook of `attention-retention-work` and of `attention-service` (the stop of the test database) timed out at 30 s while other agents loaded the machine. No assertion failed. CI runs both files. - `tsc --noEmit` for the server package: no errors. The new test files and the script are outside the server `tsconfig`; a focused `tsc` program for them reports no error in these files. - `pnpm check:module-boundaries`, `node scripts/check-no-git-push.mjs` and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: pass. - The first CI run of this pull request failed `ci / Runner support suite`. An earlier revision also let the eligibility functions of `packages/shared` accept a prebuilt index, and the source of `packages/shared` is pinned for the runner SDK source proof (`tests/runner-e2e/source-sdk-public-inputs.json`). The fix does not need that change, so the current revision leaves `packages/shared` as it is on `master`: no pin moves, and the UI, the CLI and the runner use the same eligibility code as before. `source-sdk-public-inputs.test.mjs`, `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` pass locally on this revision (58 tests), and the drift guard reports no stale pin. - `pnpm verify` returns exit 2 (broader verification required) because the change adds a file under `server/scripts/`. CI supplies the broad coverage. The repo-wide typecheck, build and test suite were not run locally (shared machine). Not verified: - The effect on the hosted instance. This pull request does not deploy. The local requests take 0.1 s to 0.4 s, and the hosted requests took 15 s. Thus most of the hosted time is not this code: the hosted main thread is saturated by other work, and each of the 69 to 85 statements of a summary request waits for its turn on that thread. This change removes CPU time and allocation from that thread. It does not remove a statement. - The number of agents and of stalled reviews of the hosted company. The hosted allocation sample has most of the bytes in `classifyIssueReviewPaths`, which is the cost for each review (reviews × agents). The local fixture has more stalled reviews, where the owner candidates cost more. - A remaining cost inside the classification: each eligibility decision of `@paperclipai/shared` indexes the agent list again. One classification now does this one time for each agent (agents² map entries for one request, 3 ms with 180 agents), not one time for each check. The helper can accept a prebuilt index in a later change; that change moves the SDK source pins, so it needs its own review. - A second, independent cost that this pull request does not change: `GET /attention/summary` returns counts only, but it builds the full feed. After this change, the largest application blocks in its profile at 9000 tasks are the full task list of the 900 blocked tasks (15 ms on the main thread, of which 9 ms decode description previews that the summary never returns), the blocker attention of those tasks (9 ms), and the text of its 85 statements. A later change can give the summary a count-only path. ## Risks - The attention feed decides what a person is told needs them. The risk is a classification that differs from the previous one. Mitigation: the rules did not move. Only the lookups changed (a map lookup instead of a scan, a stored eligibility instead of a new one). The oracle tests compare the findings and the paths for random graphs, for a named graph and for every classification of the route tests. The benchmark compares the full responses. - The index is a snapshot of the input. `createIssueReviewPathClassifier` documents that the input must not change while the classifier is in use. The two callers build the input and use it in the same synchronous block. - When the input has no `now`, one classification now reads the clock one time. Before, `classifyIssueGraphLiveness` read it one time for the blocker checks and one more time for each review. All callers in the server that classify many tasks pass `now` or run in one synchronous call, so the values differ by less than the run time of the call. - The oracle file is a copy of the previous classification (about 640 lines, test-only). A later change of a classification rule must change the oracle in the same way. The header of the file says so. - The repository rule in `CONTRIBUTING.md` says that a pull request has no code comments. This pull request has comments where the code has a constraint that the names do not show (the memo key, the snapshot rule, the purpose of the oracle, the usage of the script), as the merged changes around this code do. - No migration. No query change. No change to company scope or to actor permissions: the change is after the reads, on rows that the same queries return. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests, CPU profiles). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
782d6bead4 |
Configure managed plugin Runner lifecycle (#21065)
## Thinking Path > - Paperclip runs agents in managed environments. > - An agent can use an explicit environment or the instance default. > - A lifecycle plugin must receive the driver that provisions the agent. > - A managed plugin environment also needs to select the Runner process lifecycle. > - This change gives lifecycle selection and provisioning the same environment lookup. ## Linked Issues or Issue Description **What existing behavior does this improve?** Managed plugin environment provisioning and lifecycle configuration. **Current behavior** Lifecycle calls lack the selected plugin driver. Managed plugin configuration cannot select warm Runner execution. **Proposed behavior** Resolve the environment once per lifecycle operation. Pass the driver key for the plugin that serves it. Parse and persist the managed Runner lifecycle mode. The portable ownership and workspace placement changes are in [#21066](https://github.com/paperclipai/paperclip/pull/21066) and [#20743](https://github.com/paperclipai/paperclip/pull/20743). ## What Changed - Share agent environment resolution between required plugin selection and the lifecycle RPC. - Pass `environmentDriverKey` to the plugin that serves the selected environment. - Allow managed plugin environments to set `warm` or `per_turn` and persist the selected mode. - Advertise `pluginEnvironmentRunnerLifecycleMode` in the release capability manifest so senders can check support before including the field. ## Verification - Built shared types, the plugin SDK, and Runner. Server typecheck and SDK source-pin check passed for the initial layer. - The capability contract test failed before the manifest fix and passed after it. Independent verification passed all 160 tests across managed configuration, plugin environment persistence, lifecycle routing, and environment reconciliation. - Routing coverage includes the instance default, an explicit second driver from the same plugin, and an explicit sandbox. - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/43193311862) passed on `f07b192c21`. The first attempt timed out in an unchanged workspace-recovery fixture and then collided with a background recovery action. The exact test and all 44 tests in its suite passed locally; the isolated shard retry passed without code changes. ## Risks - Existing managed configuration can omit the lifecycle mode. Senders must check the target release for `pluginEnvironmentRunnerLifecycleMode` before sending the new field. Publishing this signal does not update external deployment tools. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1af987690d |
perf(storage): hash and encrypt workspace snapshots off the server thread (#21138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process serves every request and agent turn on one thread. The same process saves each remote workspace as one encrypted archive in object storage, and restores it before a turn > - `server/src/services/remote-workspace-snapshot-storage.ts` hashed, encrypted and decrypted every byte of those archives on that thread. `server/src/storage/s3-multipart.ts` computed the Content-MD5 of each uploaded part on that thread too > - Hashes and ciphers are synchronous in Node. A CPU profile of a busy hosted server showed this work on the main thread, which was at 93% of one core > - Each megabyte of a snapshot delays every request and turn that waits behind it > - This pull request moves those passes to a small, bounded set of worker threads. The checks, their order, their failures and the stored format do not change > - The benefit is that a 1 GiB save uses about 0.7 s of the server's thread instead of about 2.8 s, and an unrelated request during the save is answered in 1.3 ms (p99) instead of 16.7 ms ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `workspace snapshot hash encrypt`, `event loop` and `archive io`. Related merged work: #19900 gave these stream passes 1 MiB buffers, #20554 removed a read-back from migration, #20722 took runner artifact hashes off the event loop, #20709 records long event-loop blocks. No open pull request covers this change. **What happened?** A 30 second CPU profile of the main thread of a hosted server (commit |
||
|
|
5b698b68cf |
perf(run-events): append a native event in one transaction of two statements (#21131)
One appended native run event cost eight statements on the database connection (BEGIN, SAVEPOINT, run lock, replay probe, sequence update, insert, count/max read, COMMIT). The query builder built five of them again for every event, and each event created two transaction objects. A native source event now takes the run lock and then runs one statement that finds a replayed event or reserves the sequence and inserts, and reads the retained count and maximum for the acknowledgement. The lock stays a statement of its own so the append reads a snapshot taken after the lock was granted. The control-plane port passes its controller check to the append instead of opening an outer transaction, so there is no SAVEPOINT. The hot statements are fixed SQL text. Events that wake a queue still register the wake before the write and only when the event is new. Order, sequence numbers and durability do not change: every event still commits before its append returns. Local benchmark (server/scripts/benchmark-run-event-append.ts), one native event: 8.3 -> 4.6 statements, 589 -> 222 KB allocated, 1.09 -> 0.53 ms CPU, 28 -> 11.5 garbage collections per 1,000 events. One ordinary event: 105 -> 27 KB allocated. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
9e910ff9ae |
fix(chat): keep raw IDs and internal codes out of chat replies (#21081)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and the agent replies in plain
text
> - In a full chat-eval cohort, 47 of 158 passing replies showed raw
UUIDs, error codes, field keys, tool names or server paths
> - Users must not see internal jargon or raw IDs, so these replies are
a product defect, even when the facts are correct
> - Agents copy these terms from API error messages and saved fields;
the chat directive did not tell them to translate
> - This pull request fixes the sources: the chat directive, and API
denial messages that put IDs and policy codes in the human message
> - The benefit is plain replies for every model, while machine codes
stay available in separate fields
## Linked Issues or Issue Description
**What happened?**
Chat replies quote internal names and raw IDs. Examples: run, handoff,
revision, interaction and trigger UUIDs in prose;
`native_session_retry_exhausted`, `board_access_required`,
`review_policy_denied`, `local-board`, `enableCases`; a raw user ID
inside the inbox denial message.
**Expected behavior**
Replies name records the way the person sees them (task identifier,
title or name, with a link) and explain codes in plain words. IDs stay
only in link targets, in commands the person must run, or when the
person asks for an ID.
**Steps to reproduce**
Run the chat-features OpenCode cohort (for example `cases.close-case`,
`routines.webhook-trigger`, `inbox-search.archive-without-grant`) and
read the agent replies.
**Paperclip version or commit**
`master` at
|
||
|
|
8bdd53c11f |
perf(server): start no git process for server info outside a checkout (#21122)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server reports which commit it runs. `GET /api/health` and `GET
/api/liveness` return it, and the board UI and the hosting control plane
poll these routes
> - The commit comes from `getServerInfoSnapshot()` in
`server/src/server-info.ts`. The health route calls it on each request.
The snapshot keeps its Git metadata for 3 seconds, then reads it again
with `execFileSync("git", ["show", ...])`, which is a synchronous child
process
> - A container or packaged install has the `git` command but no
checkout. There `git show` can only fail, and the server then uses the
commit that the build supplied. The failed start still stops the event
loop for the length of a process start
> - A CPU profile of a busy hosted server showed this cost on the main
thread
> - This pull request looks for a checkout before it starts `git`.
Without a checkout the server takes the build-commit path directly
> - The benefit is that health polls start no child process on a server
that does not run from a checkout, and the responses stay the same
## Linked Issues or Issue Description
No public issue exists. I searched open and closed pull requests for
`server-info git health`, `readGitInfo` and `getServerInfoSnapshot`.
Related merged work: #8753 added the 3 second refresh so that the debug
view follows a checkout, #10563 and #10566 added the build commit,
#18649 took the process start time off the Git probes. No open pull
request covers this change.
**What happened?**
`server/src/app.ts` creates the health routes without a `serverInfo`
option, so each `GET /api/health` request calls
`getServerInfoSnapshot()`. When the 3 second cache is expired, the call
starts `git show -s HEAD`. On an install without a checkout the command
fails each time, so the failure is paid again on each refresh. The rate
is one `git` start for each request when requests are 3 seconds or more
apart, and one start for each 3 seconds when requests are more frequent.
It is not one start for each request in a burst.
A 30 second CPU profile of the main thread of a hosted server (commit
|
||
|
|
ef2b53b675 |
fix(environments): keep resuming an idle sandbox whose resume timed out (#21113)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on the next wake > - When the resume call times out, the run failed after 90 seconds with `reusable_sandbox_resume_failed (cause=transport_timeout)` > - The provider keeps starting the sandbox after the host call times out, so the sandbox is usually ready a little later > - The run gave up after one call, and the task then waited behind a Retry button > - This pull request keeps the same run retrying an idle sandbox whose resume only timed out, up to the provider's start deadline > - The benefit is that recovery works automatically and a sandbox with unsaved work is never destroyed ## Linked Issues or Issue Description Refs #20626, #20828, #21020 **What happened?** Three Codex runs on Daytona failed before provider dispatch with: `Reusable sandbox lease "<id>" could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` Each run had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Each run took 100 to 122 seconds: one resume call used the whole 90-second budget, so no in-run retry was possible. The preserved leases were idle. Their last runs had ended about two hours before, with a confirmed stop-and-retain. The workspaces were not safe to replace, so the run failed strictly. The automatic bootstrap retry was then refused ("The automatic retry could not revalidate the original user continuation."), and all three tasks ended `blocked` behind a Retry. **Expected behavior** A resume that only timed out, before any provider work, on a sandbox that no live run holds, continues in the same run. The run keeps the same sandbox and does not create a replacement or destroy the sandbox. **Steps to reproduce** 1. Let a task's reusable Daytona sandbox stop and stay idle for some hours (it auto-archives after 60 minutes). 2. Retry the task. The workspace has unsaved work, so replacement is not safe. 3. The `environmentResumeLease` call takes longer than 90 seconds. The host call times out (`RPC call "environmentResumeLease" timed out after 90000ms`), the 90-second budget is spent, and the run fails with `reusable_sandbox_resume_failed`. **Paperclip version or commit** Staging at `849f109f` (current `master` has the same code path). **Deployment mode** Hosted staging (cloud), Daytona sandbox provider plugin, reusable leases on. **Agent adapter(s) involved** `codex_local` (Codex) on a Daytona sandbox environment. The change is adapter-independent. **Root cause** `server/src/services/environment-runtime.ts`, reusable resume loop: the first `environmentResumeLease` call got the whole 90-second budget as its RPC timeout. When that call timed out, the loop saw no time left and threw. The 90-second budget assumes that the bounded bootstrap retry resumes the same sandbox later. That retry is not always admitted. A timeout is different from a refusal: the provider is still starting the sandbox. ## What Changed - `reusable-sandbox-resume-replacement.ts`: add `reusableSandboxLeaseIsIdle` (lease is `released` or `retained` and no other live run holds it). `reusableSandboxReplacementIsSafe` now uses it, with the same rules as before. Add `REUSABLE_SANDBOX_IDLE_RESUME_BUDGET_MS` (330 seconds, the Daytona start deadline plus the call margin). - `environment-runtime.ts`: when the 90-second budget ends on a `transport_timeout` and the preserved lease is idle, extend the deadline one time, up to the provider start deadline (at most 330 seconds from the first attempt). The run keeps retrying the same sandbox with backoff. Each call still has at most 90 seconds. Refusals, a live owner, and other failures keep the current behavior. The run never replaces or destroys the sandbox because of this extension. - `doc/execution-semantics.md`: document the idle-timeout extension. ## Verification - Red before the fix, with the new tests in `server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts` (staging shape: released lease, finished owner run, workspace with unsaved work, 300-second provider deadline, host RPC timeout): 2 tests failed with the staging error `could not be resumed (cause=transport_timeout); the lease was preserved and no replacement was created.` - Green after the fix: 16/16 pass. The timed-out idle sandbox resumes in the same run with no create or destroy call. If every call times out, the run stops at or before 330 seconds, makes more than one call, and keeps the lease `released`. A sandbox that a live run still holds gets one call and fails at 90 seconds, as before. - `pnpm exec vitest run src/__tests__/environment-runtime.test.ts src/services/environment-runtime-reusable-lease-probe.test.ts src/services/remote-workspace-transport.test.ts`: 378 pass. - `tsc -p server/tsconfig.json --noEmit` passes. - `pnpm verify` exits 2 because a doc is a global CI input. Required CI supplies the broad coverage. - Not run: a live Daytona resume on staging. No later run attempted the three preserved leases, so it is inferred, not observed, that the sandboxes were ready after the host call timed out. ## Risks - A run can now wait up to 330 seconds in preparation before its first turn, but only when the resume timed out on an idle sandbox. Before this change, that run failed and the task waited for a person. - A provider that is down (not slow) now takes up to 330 seconds to fail instead of 90 seconds. The failure is the same `reusable_sandbox_resume_failed`, and the lease is still preserved. - No persisted fields change. Leases saved by older releases follow the same rule. - The refused bootstrap retry admission for task Retry continuations is a separate gap. This pull request does not change it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
704b3f4d63 |
fix(server): a card accepted while its run settles keeps one continuation (#21094)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run ends while its task is in review, Paperclip checks that the task still has a maintained review path. If it has none, Paperclip queues one review-path recovery run > - An approval card for a tool action sends its continuation through a delivery receipt. The receipt waits until the source run has settled and the action has finished > - A person accepted a card while the source run was still settling. The interaction was no longer pending, and the continuation wake did not exist yet, so the review-path check found no path and queued a recovery run > - The real continuation then ran too, so the task got two follow-up runs for one decision > - This pull request counts an undelivered tool review continuation as a maintained review path > - The benefit is that a card accepted during settle produces exactly one continuation run ## Linked Issues or Issue Description **What happened?** An agent posted a tool approval card and its run finished. The person accepted the card 1.3 s later, while that run was still settling. The review-path check saw no pending interaction and no queued wake, and it queued an `issue_review_path_lost` recovery run. The accept continuation was deferred behind that run and then started as a second follow-up. **Expected behavior** The accepted card's pending continuation counts as the task's review path. Exactly one continuation runs. **Steps to reproduce** Have an agent in a task that is in review post a tool approval card. Accept the card while the creating run settles and before the tool action delivers its result. Before this change, a review-path recovery run was queued in addition to the continuation. ## What Changed - `server/src/services/issues.ts` (`listIssueReviewAttentionMap`): reads tool review continuations for the review tasks in one statement. A continuation counts as a queued review path when (a) its receipt is undelivered, its card is resolved, and the requesting assignee can still receive it (not paused, terminated or pending approval), or (b) its delivered continuation wake, or that wake's run, is still active. The path list leaves out a continuation whose wake or run is already listed, so each continuation shows once. The path shows as "Queued tool action review wake". - `heartbeat-issue-liveness-escalation.test.ts`: a new embedded-Postgres test runs a heartbeat for a task in review with an accepted card whose tool action is still executing. It checks that no review-path recovery wake is queued for that run. It then completes the action, delivers it, and checks that exactly one continuation wake exists. ## Verification - The new heartbeat test failed before the change (one `issue_review_path_lost` wake) and passes after it. - A new parameterized attention test covers an undelivered receipt, a delivered continuation (listed and not listed in the wake list), a pending card, a terminated assignee and a consumed continuation. The terminated, pending-card and delivered cases were red against the earlier revisions of this change. - `pnpm vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts src/__tests__/issue-review-attention.test.ts src/services/recovery/review-path-recovery.test.ts src/__tests__/attention-service.test.ts src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts src/__tests__/tool-gateway-service.test.ts`: 332 passed. - Product evidence: a paid chat-eval run of `questions-and-approvals.approve-tool-once` on a branch that combines this change with three related fixes. The result is in a PR comment. ## Risks - Every caller of review attention (review-path recovery, its escalation, the attention feed and the approvals view) now sees a pending tool review delivery as a covered path. That is the intended state: the delivery service always sends one continuation for it, and its periodic sweep retries a failed delivery. - One more indexed query for each review-attention read. It reads only undelivered receipts for the given tasks. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
81d2e4d8d9 |
fix(native): end a run whose finalizer a newer run superseded, so it stops holding its task (#21106)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run saves its accepted result, then a finalizer exports the workspace and applies the result to the task. > - When a newer run already committed the task's decision, the finalizer retires the older result as `native_finalization_superseded`. > - That retirement set the finalizer terminal, but it did not end the run. The run stayed `running` and held the task and every run queued behind it, with no way out except a person. > - A stuck run must never hold a task and its queue forever. > - This pull request ends the run when its finalizer is retired as superseded, and it ends runs that an older release left in that state. > - The benefit is that the task continues with its queued runs by itself. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, a native run accepted its result at 11:31:08Z. The workspace export then failed (`remote_workspace_archive_fs_eacces`). The task's last decision came from another run that was created 42 seconds later but ran earlier. The finalizer judged this run superseded: it set the finalizer to `terminal_failure` with `native_finalization_superseded`. The run stayed `running` with "Recovery needed". Over one hour later it still held the task, and five queued runs waited behind it. A server upgrade at 12:02Z did not change it. **Expected behavior** A run whose finalizer is retired ends. The task keeps the newer run's decision, the retired result is never applied, and the queued runs proceed. **Steps to reproduce** 1. Accept a native result for a run. 2. Commit a status decision on the same task from a run that was created later. 3. Let the older run's finalization fail once, so the finalizer records the failure. 4. See that the finalizer is `terminal_failure` / `native_finalization_superseded` and the run is still `running`. **Root cause** `recordRetryableFailure` (`server/src/services/native-runtime/native-run-finalizer.ts`) projects a terminal run status only when `exhausted && !supersededByNewerRun`. A superseded retirement therefore never ended the run. No other path ends a run whose finalizer is already `terminal_failure`: the reconciler skips `terminal_failure` rows. **Paperclip version or commit** Staging Core `3eca7a6112`. The behavior is the same on master `482063aeca`. ## What Changed - `native-run-finalizer.ts`: a superseded retirement now ends the run with the same status mapping as an exhausted finalization (workspace failure: `failed`; otherwise the provider's terminal state). It does not change the task, does not apply the result, and does not keep the workspace export lease. The run's error says in plain words that a newer run already decided the task. - `native-run-finalizer.ts`: add `endSupersededNativeFinalizationRuns`. It ends a native run that is still `running` while its finalizer is `terminal_failure` / `native_finalization_superseded`. This covers rows that an older release wrote. It locks in the finalizer's order (coordinator, task, run), and re-checks the row under the lock. - `native-run-projection-work.ts`: the registered native run projection worker (it loads at startup, pages through all candidates, and wakes on finalization changes) also selects such runs and ends them with `endSupersededNativeFinalizationRuns`. - `doc/execution-semantics.md`: document the rule. - No schema change. Nothing changes on the path that starts a run or a turn. ## Verification - `server/src/__tests__/native-finalization-recovery.test.ts`, red on master and green on this branch: - "retires an older failed finalizer when a newer run already committed the issue": the retired run now ends (`succeeded`, `finishedAt` set). Master: the run stays `running`. - "ends a run an earlier release left running after retiring its finalizer as superseded": the shape of the staging row (workspace export failed, superseded finalizer, run `running`). The repair ends it as `failed` with the plain message. The task, its decision and the finalizer stay unchanged, and no status decision is written for the run. A second pass changes nothing. - `server/src/__tests__/native-run-projection-work.test.ts`, "ends, at startup, a run an earlier release left running after retiring its finalizer as superseded": the registered worker ends the staging-shaped row at startup and settles its claimed wake. Red without the worker change. - Related suites pass: `native-finalization-recovery`, `native-run-projection-work`, `native-finalization-reconciler-timing`, `native-reconciliation-timing`, `native-workspace-finalizer`, `native-workspace-finalizer-failure`, `native-run-finalizer-telemetry`. - `tsc --noEmit` for `server`. ## Risks - Low risk. The change only ends runs whose finalizer is already terminal and superseded. It never applies a result and never changes the task. - A superseded run whose workspace export failed now releases its sandbox lease through the normal settlement instead of waiting forever. Its result was not applied, because a newer run decided the task. - Not changed here: the superseded check compares run creation times. A run created earlier but executed later is still judged superseded. This PR makes that verdict end the run; it does not change the verdict. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a2f618cae |
fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for a few minutes, then the old server gets SIGTERM and hands its native runs to the next controller. > - #21046 taught the legacy reaper to leave a run with its owner while the owner's server process is proven alive. Native runs have a similar gap. > - Runtime selection writes a native run's coordinator with no controller and no lease, and stops the run lease. The executor claims the coordinator about 20 seconds later. In that window the other server's sweep sees no owner and marks the run "ownership could not be verified". > - When the old server then hands the run off before any Runner checkpoint exists, no sweep looks at it again. Only the next server boot does, 15 to 40 minutes later. The task and its queued runs wait. > - This pull request leaves such a run with its live preparing server, and lets the serving server re-adopt a handed-off run through the ordinary restart claim. > - The benefit is that a run started during an upgrade continues within one sweep, with no false "blocked" mark and no wait for another upgrade. ## Linked Issues or Issue Description Refs #21046 **What happened?** On a hosted staging server, an upgrade started a new server at 12:19:57Z. The old server started a native run and selected its runtime at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership could not be verified; automatic recovery is blocked" (`observed_owner_unverified`). The old server's executor claimed the run at 12:20:51Z. The old server got SIGTERM and suspended the run at 12:20:54Z (`awaiting_runner_reattach`). After that the run showed "Confirming execution" for 29 minutes, and two queued runs waited behind it. At 12:49:45Z the next upgrade's startup recovery restarted its bootstrap, and the run succeeded at 12:58Z. **Expected behavior** - A server does not mark a run unverified, or recover it, while another live server is preparing it. - A run that a graceful shutdown hands to the next controller is re-adopted by the serving server at once, not at the next boot. **Steps to reproduce** 1. Seed a native run whose coordinator is `observed` with no controller and no lease, and whose `controllerProcess` names another server with a live controller instance lease. 2. Run the recovery sweep on this server. See that the run gets `native_execution_ownership_unverified`. 3. Claim the coordinator for the other server, then write what its graceful shutdown writes (`restart_suspended`, `awaiting_runner_reattach`, lapsed lease) and stop its instance lease. Give the run a predecessor's Runner PID. 4. Run the sweep again. See that nothing claims the run. **Root cause** - `recoverActiveExecutions` (`server/src/services/heartbeat/recovery.ts`) treats an `observed` coordinator without a live controller lease as an owner it cannot verify. Between runtime selection and the claim, the only owner on record is the run's `controllerProcess`, and the sweep did not check it. - `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`) selects runs for the restart claim only when they have a controller checkpoint, or no process identity at all. A run handed off just after its claim has no checkpoint, and a follow-up carries its predecessor's Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so only a server boot looked at them again. **Paperclip version or commit** Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server). The behavior is the same on master `849f109f62`. ## What Changed - `recovery.ts`: before it lists expired native runs, the sweep checks runs in the runtime-selection window (coordinator `observed`, no controller, no lease, not tracked here). When the run's `controllerProcess` names another server process that the shared owner prover proves alive, the sweep leaves the run with that server: it does not recover it and does not mark it unverified. It re-checks within 30 seconds. After 5 minutes without a claim (`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the existing rules apply. - `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that proof. - `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities` also selects a run that a graceful shutdown handed off and whose lease lapsed: recovery state `awaiting_runner_reattach` or `awaiting_evidence`, with a `restart_suspended` history entry that no restart claim followed. It stays selected through ownership waits (the departing server can still be alive at the first look). `recovery.ts` schedules a 30-second re-check for an ownership wait that has no retry of its own. The ordinary restart claim then proves the takeover and picks the disposition, exactly as the startup pass of a new server does. A row that an older release already marked unverified is re-adopted too: the claim clears the mark. - `doc/execution-semantics.md`: document both rules. - No schema change. Nothing changes on the path that starts a run or a turn: the new proof runs only in the recovery sweep, and only for runs in the window. ## Verification - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, block "a run the old server started during an upgrade overlap". Red on master, green on this branch: - "leaves the run with its live preparing server, then re-adopts it after that server hands it off": two concurrent sweeps leave the run `running` with no error and do not touch the coordinator. After the old server claims, hands the run off and stops, one sweep of the new server re-adopts it and reaches provider dispatch. Master: the run is marked `native_execution_ownership_unverified`, and after the handoff nothing claims it. - "keeps a handed-off run eligible while the departing server is still alive, and re-adopts it once it exits": the first sweep waits (`awaiting_evidence`), the sweep after the departing process exits re-adopts the run. Red with a filter that matches only the last history entry. - "re-adopts a handed-off run that an older release already marked ownership-unverified": the stuck staging shape. Master: nothing claims it. - Related suites pass: `heartbeat-native-restart-suspension` (all 109), `services/heartbeat/recovery.test.ts` (all 103). - `tsc --noEmit` for `server`. ## Risks - A run that another live server prepares, but never claims, now stays `running` for up to 5 minutes before the existing rules apply. That is the same bound #21046 uses. - The guard trusts the shared owner prover. It reads "alive" only from a live controller instance lease or a live PID in the same PID space. - More handed-off runs reach the restart claim earlier. The claim keeps all its fencing and evidence checks; this change only makes the serving server ask the same question a new boot asks. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a65644d1e1 |
fix(retry): continue an accepted connected Retry from recovered saved work when its chat replay is refused later (#21080)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks from a connected chat (for example GitHub pull request reviews) show Retry when a run fails > - A person's Retry on such a task first asks the chat retry owner to repeat the exact failed chat request. #21038 made a refusal at the click continue the task from saved work instead > - The chat owner can also defer its source check (a busy identity, endpoint, or agent row lock) and refuse it a few seconds later. Then the route has already answered 202, and the task Retry receipt was marked failed with no run > - The task stayed blocked, and its execution blocker told the person to send a message, while the interruption projection still offered Retry > - This pull request makes that late refusal take the same board continuation as a refusal at the click, and makes the blocker offer Retry for recovered saved work > - The benefit is that an accepted Retry always starts work, and the two projections agree ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, about 20 blocked tasks of one busy GitHub review agent (`paperclip_runner`, native remote runs) did not restart after Retry. `POST /api/issues/:id/retry` returned 202 with `accepted: true`, a `chatActionId`, and `presentation.state: retry_scheduled`. No run was created. The task Retry receipt changed to `failed` 5 to 10 seconds later, and the task stayed blocked. The activity record of each click (`chat.failed_run_retry_requested`) has `sourceAuthorizationPending: true`, so the chat owner deferred its source check at the click. No chat wake row was written, so the chat replay ended before admission. The last run failed with `native_continuation_requires_reconciliation` / `runner_remote_process_identity_unavailable`. Its saved work was recovered (`workspaceRecovery.status: recovered`, remote lease destroyed with a termination receipt). The projections disagreed: `interruption` had `canRetry: true`, but `executionBlocker` had `canRetry: false` and `nextAction: "Saved work recovered. Send a message to continue."`. Root cause: when the chat retry action settles `failed`, `releaseUnadmittedTaskRetryInput` in `server/src/services/chat-channels/task-retry-input.ts` marked the linked task Retry receipt `failed`. #21038 added a board-continuation fallback only for a refusal inside the click transaction. A refusal after a deferred source check (for example 8 lock-busy attempts on the agent row, about 7 seconds) had no fallback. Separately, `getExecutionBlocker` hid Retry for every chat-bound task (`canRetry && !chatBinding`), and `workspaceRecoveryDisplay` always said "Send a message to continue" for recovered work. **Expected behavior** A Retry that is accepted starts work. If the exact chat replay is refused after the click, the person's Retry continues the task from its recovered saved work. The execution blocker and the interruption both offer Retry, and neither tells the person to send a message to unblock a failed run. **Steps to reproduce** 1. Bind a task to a GitHub chat conversation. Make its latest native run fail with `native_continuation_requires_reconciliation` and recover its saved work. 2. Hold the chat identity advisory lock while you click Retry, so the source check is deferred (`sourceAuthorizationPending: true`). 3. Hold a row lock on the agent while the chat retry processes, so the deferred check fails 8 times. 4. The route returned 202. The chat action settles `failed`, the task Retry receipt changes to `failed`, and no run starts. **Paperclip version or commit** `master` at |
||
|
|
849f109f62 |
fix(server): agent inbox reads now apply private-task visibility (#21092)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents read a person's Mine list with `GET /api/agents/me/inbox/mine`, and their own queue with `GET /api/agents/me/inbox-lite` > - The company task list routes apply private-task visibility for the reading actor > - The two agent inbox routes should use the same visibility rules, so that each route returns the same tasks for the same actor > - This pull request makes agent inbox reads apply private-task visibility > - The benefit is one consistent visibility rule across task list routes ## Linked Issues or Issue Description **What existing behavior does this improve?** Agent inbox reads now apply private-task visibility. **Subsystem affected** Server API: agent inbox routes. **Current behavior** The agent inbox routes build their lists without the issue read condition that the company task list routes use. **Proposed behavior** The agent inbox routes use the same issue read condition and per-row read decisions as the company task list routes. **Reason and benefit** All task list routes apply one visibility rule. ## What Changed - `server/src/routes/agents.ts`: `GET /api/agents/me/inbox/mine` adds `issueReadSqlCondition` to its list query and keeps only the rows that `access.decideReads` allows. - `GET /api/agents/me/inbox-lite` adds the same read condition to its list query. - `inbox-mine-routes.test.ts`: a new embedded-Postgres test for the agent Mine read with private and open tasks. - `agent-permissions-routes.test.ts`: the mocked route test expects the read condition and the read decisions. ## Verification - `pnpm vitest run src/__tests__/inbox-mine-routes.test.ts`: 7 passed. The new test was red before the route change and is green after it. - `pnpm vitest run src/__tests__/agent-permissions-routes.test.ts src/__tests__/openapi-routes.test.ts`: all passed. - A chat eval run on a branch that combines this change with three other fixes. The result is in a PR comment. ## Risks - Low risk. The routes return only tasks that the reading agent can open through the task routes. - The Mine route adds one batched read decision for each request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
67403baaa4 |
fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first
## Linked Issues or Issue Description
Refs #20186 (the same treatment for a governance pause Stop)
**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.
**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.
**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.
## What Changed
- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.
## Verification
- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.
## Risks
- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6d8f59d494 |
fix(api): name the secret binding proposal as the way to give an agent a company secret (#21090)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents find product API routes through the runner API search,
which ranks operations by their OpenAPI summary and description
> - In the chat eval `secrets.configure-company-secret`, a person asked
the CEO to let Morgan use an existing deployment API key
> - The agent searched for agent configuration and environment variable
routes, tried `PATCH /api/agents/{id}` (403) and spent 3.5 minutes
before it used `POST /api/agents/me/secret-proposals`
> - The summary "Propose a company secret or agent secret binding" did
not match how agents describe the task
> - This pull request rewrites the summary and adds a description that
names the binding fields
> - The benefit is that the first search finds the supported route, so
the turn is short
## Linked Issues or Issue Description
**What happened?**
Runner API search did not rank `POST /api/agents/me/secret-proposals`
for the queries an agent uses to give another agent a company secret
("agent configuration update env secret", "agent environment variables
set secret ref"). The agent probed other routes for 3.5 minutes.
**Expected behavior**
Those searches return the secret binding proposal route in the first
results, and the result explains `secretId`, `configPath` and
`targetAgentId`.
**Steps to reproduce**
Call `searchRunnerApi({ query: "agent configuration update env secret",
limit: 3 })` on master. The proposal route is not in the results.
## What Changed
- `server/src/routes/openapi.ts`: new summary and description for `POST
/api/agents/me/secret-proposals`. The description states the
reporting-line target rule, the `env.<KEY>` config path, and that a
board member approves the binding.
- `server/src/__tests__/secret-binding-discovery.test.ts`: the four
queries return the route in their top three results. The test fails on
master (red) and passes with this change (green).
## Verification
- `npx vitest run` on 24 server guidance, discovery, OpenAPI and runner
API test files: 2305 passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid chat eval run: see the PR comment.
## Risks
- Low. Only OpenAPI text changes. Other search results can move down by
one place for queries that share these words.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b853cc28ae |
perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server process starts, recovers saved work, and then reports ready on `/api/ready`. The deploy health gate waits for that signal. > - On a busy staging workspace, one boot took 85 s to bind the listener and 4.4 minutes to report ready. Small, idle stacks take 20–90 s. > - Two parts of that time do not need to block readiness. The server published several hundred UI assets one at a time before it bound the listener. After the listener bound, it waited for the first pass of each periodic recovery queue, and those passes grow with the backlog. > - The volume-free data gate, which can refuse a start, ran last. By then the server had already dispatched queued runs that a refusal would abandon. > - This pull request moves the refusal gate to the start, publishes the assets 8 at a time with replayable bodies, and runs the backlog-sized first passes after readiness, in their original order. > - The benefit is a faster time from container start to ready, and no work admitted before a gate that can refuse to start. ## Linked Issues or Issue Description No public issue exists. This is the problem description (enhancement template). **Is your enhancement related to a problem? Please describe.** Server boot time grows with the saved backlog. These timestamps come from the log of one staging boot (release |
||
|
|
f8e1fb0278 |
perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha |
||
|
|
db8848f559 |
fix(heartbeat): bound the I7 queued-run check with a resumable cursor (#21022)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20986 added a reconciler check that reports a queued run whose dispatch signal was lost (I7 `queued_run_not_dispatched`), and a worker that re-drives such runs. > - The Ponytail review of #20986 (4/5 on `fbab23cce6`) found three issues. #20986 merged at that head before the fix commit was pushed. > - The new I7 check read every queued run with no cap, while the report still said that each I7 check reads at most 2,000 records. > - This pull request adds that fix commit to master. The check now reads a bounded slice per pass and continues from a cursor. The page cursor no longer depends on the session date style. An agent with a scheduled retry now counts as busy. > - The benefit is that the reconciler pass stays bounded when many runs are held on purpose, and the coverage statement is true again. ## Linked Issues or Issue Description Refs #20986 Follow-up to the Ponytail review of #20986: https://github.com/paperclipai/paperclip/pull/20986#pullrequestreview-5868879754 (review of `fbab23cce6`, score 4/5). **What is wrong?** 1. `checkQueuedRunsDispatched` (`server/src/services/recovery/ownership-invariant-report.ts`) paged through every undispatched queued run with no cap. `COVERAGE_GAPS.record_cap` still said each I7 check reads at most 2,000 records per pass. 2. The page cursor cast `created_at::text` back with `::timestamptz`. That works only with the default `DateStyle`. 3. A queued run on an agent with a `scheduled_retry` run was reported and re-driven, but `checkVisibleNextAction` already treats `scheduled_retry` as an active run. **Expected behavior** The I7 queued-run check reads a bounded slice per pass and reaches every run over later passes. The cursor format does not depend on session settings. A scheduled retry counts as the agent's next action. ## What Changed - `server/src/services/queued-run-dispatch-liveness.ts`: new `readUndispatchedQueuedRuns`. It reads from an optional cursor until the set ends or `maxRuns` rows were read, and returns the next cursor. `listUndispatchedQueuedRuns` uses it with no limit. The cursor text is `to_char(created_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`, which keeps microsecond precision and does not depend on `DateStyle`. The live-run exclusion now also covers `status = 'scheduled_retry'`. - `server/src/services/recovery/ownership-invariant-report.ts`: the I7 queued-run check reads at most `MAX_RECORDS_PER_KIND` (2,000) per pass. It keeps a keyset cursor in `OwnershipReportState.queuedRunCursor`, and the next pass continues from that cursor. A pass that stops at the cap keeps a follow-up pass due (`retryNeeded`), like the I3 and I4 backlogs. `COVERAGE_GAPS.record_cap` now describes this check. - `server/src/__tests__/ownership-reconciliation-report.test.ts`: new test. Bounded slices resume from the cursor and end with `next: null`. A `scheduled_retry` run on the agent removes its queued runs from the set. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/heartbeat-held-comment-wake.test.ts`: 75 tests pass on this branch (fresh master + this commit). - Server `tsc --noEmit` passes. - The restart integration test from #20986 ("starts a continuation that queued behind its predecessor's sandbox release after a graceful restart") passed with this change on the #20986 branch, in both cases (signalled and unsignalled). - Restart-chaos CR1 360 s, Codex W2, `--daytona-warm --ephemeral-server`, seed 9512 (`w-20261112T040314-s9512`, on the #20986 change set before this commit): the fault crossed the task and every rule passed (R1–R6, T, U1–U5). ## Risks - Low risk. The report reads less per pass. A very large held queue is now covered over several passes instead of one. - A queued run behind a `scheduled_retry` is no longer re-driven by the backstop. The scheduled retry's own promotion signals dispatch. - No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ad2d123dad |
fix(recovery): count a current response-wake wait as a visible next action in I7 (#20994)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The ownership reconciler checks invariant I7: an agent-assigned `in_progress` issue must have a visible next action. A live run, a pending wake, an open recovery action or a saved monitor deadline counts. > - #20935 added a parked wait. When a native agent yields `response_wake` and no response source is admitted, the arbiter commits `response_wake_waiting_for_input`. The task stays `in_progress`, and the run shows "Waiting for a reply". > - I7 did not know this wait. A task that really waits for the board was reported as `no_owner_no_wake`. > - This pull request makes I7 count that wait as a visible next action while it is still current. I7 still reports the issue when the wait is stale. > - The benefit is that a genuinely parked task is not reported as a silent stall, and a stale wait is still reported. ## Linked Issues or Issue Description **What happened?** The ownership reconciler reported `I7_visible_next_action` / `no_owner_no_wake` for an issue whose latest applied decision was `response_wake_waiting_for_input`. That is the visible "Waiting for a reply" state from #20935. **Expected behavior** A current, committed response-wake wait is a visible next action. The issue is reported again when a user comment arrives after the waiting run started, or when that decision is no longer the issue's latest decision. **Steps to reproduce** 1. Commit a native run whose arbiter decision is `response_wake_waiting_for_input`, and make it the issue's latest decision. 2. Wait past the 5-minute next-action grace window. 3. Run an ownership reconciliation pass. Before this change, I7 reports the issue. **Additional context** Follow-up to #20989 and #20935. The conditions match `hasCommittedNativeResponseWakeWait`, which recovery already uses for the same wait. Refs #20935 Refs #20989 ## What Changed - `server/src/services/recovery/ownership-invariant-report.ts`: `checkVisibleNextAction` excludes an issue with a parked response-wake wait. The wait must meet all of these conditions: - The issue's `lastStatusDecisionId` is an applied `response_wake_waiting_for_input` decision. - Its native finalization is `committed`. - Its run succeeded and belongs to the current agent assignee. - The issue has no user assignee and is not hidden. - No user comment arrived after that run started. - New test in `server/src/__tests__/ownership-reconciliation-report.test.ts`: - A current wait is not reported. - A later user comment makes I7 report the issue. - A decision that is no longer current also makes I7 report it. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts -t "response-wake wait"`. I7 reported the waiting issue. - Green: the full `ownership-reconciliation-report.test.ts` and `environment-pending-cleanup-queue-owner.test.ts` pass (43 tests). `npx tsc --noEmit -p server` passes. - Restart-chaos harness, for reviewers: the harness U1/R1 grading should treat a sample as waiting, not ownerless, when the issue's latest run shows the execution projection phase `waiting_for_answer` ("Waiting for a reply"). It should still fail when a user comment is newer than that run. The harness reads I7 through `tests/restart-chaos/invariants.ts`, so I7 changes with this PR. The U1 sampler does not read the projection yet, and this PR does not change it. ## Risks - Low risk. The change is in a report-only check. It writes nothing and changes no admission, status or recovery path. - If the conditions are too wide, a stalled task is hidden. The conditions are the same as recovery's durable-wait check, and the test proves that a stale wait is still reported. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f30005440b |
fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.
## Linked Issues or Issue Description
**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.
**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.
**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.
**Paperclip version or commit**
master at
|
||
|
|
424e2c6488 |
fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for several minutes. A legacy run proves that its owner is alive by renewing a 60-second lease on its `heartbeat_runs` row. > - #21019 added an uncharged `server_restart` retry lane for runs that a restart or replacement ended. On the first upgrade that shipped it, no run used that lane. The new server also failed many runs as `process_lost` while the old server that owned them was still serving. > - Each of these runs then spent one failure retry. A run that was already the last automatic retry was held for manual reconciliation, so the task needed a person. > - An upgrade is routine. It must not cost a task its work or a retry, and it must not leave a task waiting for a manual Retry. > - This pull request sends a run that a replacement stopped during preparation to the `server_restart` lane. It also stops the reaper from failing a run while the server process that owns it is proven alive. > - The benefit is that a task continues by itself through an upgrade, also when the database is slow during the swap. ## Linked Issues or Issue Description Refs #21019, #21042 **What happened?** A hosted staging server was replaced by an automatic upgrade. The new server started at 06:18:46Z, became ready at 06:26:17Z, and the old server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new server failed 44 runs as `process_lost` ("Process lost -- server may have restarted; retrying once", `processLossDiagnostic.controllerReplaced: true`). The old server still owned these runs and was still serving. Every reaped run was a Runner run in `executionStage: preparing` with no process recorded. Each successor was scheduled with `scheduledRetryReason: transient_failure` and `executionRetryAccounting.failureRetries: 1`. No run on the server has `retryReason: server_restart` or `executionRetryAccounting.serverRestarts`. The run evidence also shows why the leases lapsed. The old server's lease renewals waited 19 to 50 seconds on the database (`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time (`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This pull request fixes how the reaper and the retry scheduler treat these runs. **Expected behavior** - A server never fails a run as lost while the server process that owns it is alive and serving. - A run that a replacement really ended continues once, automatically, without spending the failure budget. **Steps to reproduce** 1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`, owned by another server boot, with a lapsed run lease. 2. Run `reapOrphanedRuns()` on the new server. 3. See that the successor is `transient_failure` with `failureRetries: 1`. When the chain already spent its failure budget, see that no successor is created and the run is held for reconciliation. 4. Give the owner a live controller instance lease (the old server is still serving) and run `reapOrphanedRuns()` again: the run is failed as `process_lost` anyway. **Root cause** - `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`) left every run with the bootstrap evidence `providerWorkStarted: false` to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun` gives that lane only to an `interrupted` run (or a native preparation). The reaper writes the same evidence on a `failed`/`process_lost` run that it stops during preparation. That run fell through to the charged `transient_failure` lane. The #21019 tests used runs in `dispatching`, so they did not catch it. - `legacyExecutionNeedsReconciliation` held every run whose failure budget was spent before it checked the bootstrap evidence. A run that a replacement stopped before dispatch did no provider work, but it was held for reconciliation, and the task waited for a person. - The orphan reaper (`recoverActiveExecutions`, `server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease as proof that the owner was gone. It checked the shared owner prover only for "gone", never for "alive". During the swap the old server's renewals were slow, so its leases lapsed while it was alive. **Paperclip version or commit** Staging Core `51d77648bc` (first build with #21019). The behavior is the same on master `148d2b629c`. ## What Changed - `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)` keeps the `interrupted_preparation` exception only when that lane really takes the run. `scheduleBoundedRetryForRun` passes its own verdict. Without one, only an `interrupted` run qualifies. A `process_lost` run that a replacement stopped during preparation now continues in the `server_restart` lane. - `legacy-execution-recovery.ts`: a replaced preparation (`isReplacedPreparation`) does not need reconciliation, also when its failure budget is spent. The `server_restart` bound limits it. - `recovery.ts` (orphan reaper): - When a run's lease lapsed but the shared owner prover proves its owner process alive (for example, the owner's controller instance lease is live), the reaper leaves the run with that owner. The owner stops and settles the run itself (`controller_lease_lost`, uncharged `interrupted_preparation` lane). The reaper recovers the run once the owner is gone or stopped. It also recovers the run when the owner has not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes) after the lease lapsed. - A setup that a replacement stopped before dispatch uses the `server_restart` bound for its retry decision, whatever the adapter. - `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which returns the shared prover's verdict. `legacyControllerProvenGone` uses it, and the reaper computes the proof once per run. - `server-replacement-interruption.ts`: add `isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted: false`, a recorded controller process stamp, and a server-replacement interruption. A row saved by an older release without the stamp keeps the charged lane (the existing documented rule). - `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a reconciliation hold that an older release saved on a replaced preparation is settled once its owner is proven gone, even when the failure budget is spent. The run continues in the `server_restart` lane. - `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the `legacy_interrupted_preparation_hold` kind judges a finished run's hold by its recorded controller process (`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper swaps in a random boot id, so before this change a reaped preparation's hold had owner "none" and was never settled. - `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports exhaustion from the scheduler's own `retry_exhausted` result, so a spent `interrupted_preparation` lane blocks the task even when the synchronous budget check reads an unspent restart bound. - `doc/execution-semantics.md`: document these rules. - No schema change. Nothing changes on the path that starts a run or a turn (no new waits and no file work). ## Verification - Red on `origin/master`, green on this branch (`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs ended by a server restart or replacement"): - "continues a first attempt / a run that was already the last automatic retry that a replaced server left preparing, in the restart lane". This is the production shape: a `paperclip_runner` run in `preparing`, two servers sweep at the same time. Master: `transient_failure` with `failureRetries: 1`, and no successor for the spent chain. Now: one `server_restart` continuation, due at once, with the failure count unchanged. - "leaves a preparing/dispatching run whose lease lapsed with its owner, then recovers it once the owner stops". The owner's controller instance lease is live and two servers sweep. Master: `process_lost` at once. Now: the run stays `running` with its owner. After the owner stops, it becomes `process_lost` and a `server_restart` continuation is scheduled. - "recovers the run when its live owner has not settled it within the grace". - `interrupted-preparation-hold-settle.test.ts`: "releases an older release's hold on a preparation a server replacement stopped with its failure budget spent" for an unrevoked and a reaper-revoked lease (red without the settlement and owner-stamp changes) and "blocks the task when a replaced preparation's lane is spent even though the restart bound is not" (red without the sweep change). - Related suites: heartbeat process recovery, retry scheduling, orphaned active lease sweep, recovery deadline (bootstrap and ordinary bootstrap), legacy continuation authority, undispatched execution wait, agent status recovery, legacy workspace restore recovery, legacy execution recovery, legacy controller lease, server replacement interruption, heartbeat recovery, provider failure classification, reconciliation run read, and wake-queue policy. - `tsc --noEmit` for `server`. ## Risks - A run whose owner is alive but no longer drives it now stays `running` for up to 5 minutes after its lease lapsed, and then the reaper recovers it. The owner's own sweep still recovers a run that the owner itself lost the handle for (this boot is never deferred). - The guard trusts the shared owner prover. That prover reads "alive" only from a live or grace controller instance lease, or from a live PID in the same PID space. A server whose instance lease cannot renew for 60 seconds stops itself, so a stuck owner cannot hold runs forever. - More runs continue in the uncharged lanes. The `server_restart` bound (6 in 60 minutes) still stops a loop, and the stop is shown on the task. - Rows saved by older releases: a `process_lost` row with the bootstrap evidence and an old-boot owner now continues in the `server_restart` lane instead of being held. These runs did no provider work. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only log and API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4e519bd428 |
fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending
## Linked Issues or Issue Description
Refs #21035, #21002
**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).
**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.
**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.
## What Changed
- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.
## Verification
- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on
|
||
|
|
3c058c37e6 |
fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks that come from a connected chat (for example GitHub pull request reviews) show a Retry button when a run fails > - The task projection offers Retry, but the Retry route sends a connected task to the chat retry owner, which only replays the exact failed chat request > - When that exact request cannot be replayed (the attempt already saved a result, its output was published, the latest run did not start from a chat message, or the requester's access changed), the route returns 409 and tells the person to check the connection by hand > - A Retry button that is shown must work, and recovery must not ask the person to inspect anything > - This pull request makes a person's Retry continue the task from its saved work when the exact chat replay is refused, the same way a new message on the task would > - The benefit is that the projection and the route agree, and blocked connected tasks become retryable after deploy with no migration ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, five blocked tasks that came from a connected GitHub conversation showed Retry (`interruption.canRetry: true`). Each click on Retry returned 409 `chat_failed_run_retry_not_authorized`. The message was "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state." or "This attempt already recorded a result. Send a new message on the task to continue from the saved work." The latest runs were legacy sandbox setup failures, native runs that needed reconciliation, a run that started from an issue monitor instead of a chat message, and a native run that already saved its result. Root cause: `admitTaskRetry` in `server/src/services/task-retry.ts` called the chat retry owner for every connected task and passed its exact-replay refusal straight to the person. The projection cannot predict that refusal. **Expected behavior** A shown Retry works. If the exact chat request cannot be repeated, Retry continues the task from its saved work, the same way a new message on the task would. Users do not get operator instructions. **Steps to reproduce** 1. Bind a task to a restricted chat conversation. 2. Make its latest run fail so that the chat retry owner refuses the exact replay. For example, use a run whose source is not a chat message, or a native run with an accepted result. 3. Click Retry (`POST /api/issues/:id/retry`). 4. The route returns 409 `chat_failed_run_retry_not_authorized`. **Paperclip version or commit** `master` at |
||
|
|
0dfbbdeb31 |
fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents follow the chat directive and discover API routes with search_api and call_api > - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four guidance gaps > - The missing-name rule overrode the board-only company-creation boundary; a repeated request after a decline did not post a new approval card; search_api showed the board-only question cancel route as open to agents (and withdraw as board-only); write calls carried `responseText` beside or instead of `body` > - This pull request fixes the guidance, the route authorization metadata and the pre-send refusal > - The benefit is that agents take the supported path the first time instead of asking for the wrong input, refusing a valid retry, or calling a route that returns 403 ## Linked Issues or Issue Description **What happened?** - onboarding.new-company: asked to "set up a new company for my website business", the CEO cited the missing-name rule (#21007) and asked for a business name with a question card. Creating a company is a board action, so the agent should send the person to the creation flow. - app-permissions.ask-before-action: the person declined a write card and then repeated the same request. The CEO asked the person to tell it to resubmit instead of posting a new approval card. The gateway already allows a fresh request after a new human message (`isNewHumanIntentAfterToolRejection`), but the decline text said "Do not retry the same call". - questions-and-approvals.multi-part-question: search_api listed `POST /api/issues/{id}/interactions/{interactionId}/cancel` as `board_or_agent`. The route rejects agents. The agent route, `/withdraw`, was listed as board-only although its route admits the card's creator agent and the task's assignee. - Several cases: write calls through call_api carried `responseText` with no body (for example "Received arguments: operationId, pathParams, responseText"). **Expected behavior** The agent links the company-creation flow; it posts a fresh approval card for a repeated request; search_api shows the true authorization of cancel and withdraw; a write with `responseText` is refused before it is sent, with the same guidance #21002 gave `saveResponse`. **Steps to reproduce** Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on master. Refs #21002, #21007 ## What Changed - `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the agent creates itself and never comes before a board-only boundary. For a new company the agent links [Create organization](/onboarding) (no company prefix) and repeats the business details. A repeated request after a declined approval card is an explicit retry: make the same call so a fresh card is posted. A clarification question card puts candidate links in its description, because labels show as plain text. - Decline instructions in `tool-action-delivery.ts` and `tool-gateway.ts` say that a later request from the person for the same action is an explicit retry. - OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel` is board-only and its summary names the `/withdraw` route for agents. `/withdraw` is no longer marked board-only, which matches its route. - `call_api`: a new write that carries `responseText` is refused before it is sent; the message says `responseText` never carries request fields and names the allowed body fields. Older saved calls still validate for recovery. `doc/runner-api-tools.md` is updated. ## Verification - `server/src/__tests__/agent-conversations.test.ts`: new directive tests for the board-only boundary order, the decline retry and card links. - `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`, withdraw is `board_or_agent`, the cancel summary names withdraw. - `server/src/services/native-runtime/runner-api.test.ts`: a write with `responseText` is refused before sending (three window shapes), the same write without it is sent once, and saved calls still validate. - `tool-gateway.test.ts`, `tool-gateway-service.test.ts`, `issue-thread-interaction-routes.test.ts`, `tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`: passed. - Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the eval PR: see the PR comment for results. ## Risks - A model that used to send a write with `responseText` and a valid body now gets one refusal and must resend without it. Nothing is sent before the refusal, so there is no duplicate side effect. - The directive grows by a few sentences, which adds a small number of prompt tokens per chat turn. - Agents that read `/withdraw` as board-only before now see that they can withdraw their own card; the route already enforced who may. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
940d681974 |
fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner runs keep a provider session per task. When a run stops, its session close continues in the server process > - The runner refuses every new session of the task until that close settles. A close that cannot prove the old runner suspended needs a stopped-process proof before it is released > - A reassignment stops the old assignee's run mid-turn. The new assignee's run was dispatched one second later into that gate, and it failed with `native_session_cleanup_quarantined` > - Only a Retry continued the task. Recovery must work without a person, and the needed-retry rate must be 0% > - This pull request keeps such a run queued until the close settles or its old owner is proven stopped > - The benefit is that the new run starts by itself after cleanup, without a failed run or a Retry ## Linked Issues or Issue Description **What happened?** In the paid chat eval `tasks.edit-properties` (and `tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a task while the old assignee's native run was mid-turn, with a Paperclip tool call in flight. The reassignment cancelled that run. Its session close then failed with `native_session_close_unrecoverable: runner did not durably suspend before checkpoint`, and the runner kept an operator-required cleanup gate for the task. The new assignee's run had started 1 second after the cancel. The runner's admission waited for the in-flight close. When the close escalated, admission refused the run with `native_session_cleanup_quarantined`. No run was queued again. **Steps to reproduce** 1. Run a native task and stop its run while its session close cannot settle at once. 2. Queue a run of another agent for the same task. 3. The run is dispatched into the cleanup gate and fails before it launches. **Expected behavior** The run waits for the cleanup, and starts once the close settles or the old owner is proven stopped. It never fails, and no Retry is needed. ## What Changed - `server/src/services/native-runtime/native-cleanup-admission-hold.ts`: `nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an operator-required gate it first tries `retireStoppedSessionCleanupGates`, which retires the gate when the old owner's runner and provider processes are proven stopped. This is the same proof that Retry uses. - `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an unstarted `paperclip_runner` run queued while its task has a settling or unproven close. This matches the existing `predecessor_settling_in_process` hold, which covers only the same agent. A reassignment changes the agent. - `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a released close now also wakes queued-run dispatch. A close that escalates to operator-required also wakes it, so admission tries the stopped-process proof at once. The queued-run liveness redrive covers any later proof. - `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`: an assembled heartbeat test. It is red on master (the run is claimed into a pending close and into an operator-required close). It is green here (the run stays queued, then is admitted after the close settles). A control case shows that a run for another task is not held. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3 passed; 2 red on master) - `cd server && npx vitest run src/services/quarantined-task-retry.test.ts src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-workspace-busy.test.ts src/services/queued-run-dispatch-liveness.test.ts` (180 passed) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - A run whose task has an operator-required gate that is never proven stopped now stays queued instead of failing. Before, that run failed, and Retry was blocked by the same proof. The liveness redrive re-checks it with back-off. - The hold applies only to unstarted `paperclip_runner` runs of the gated task. Other tasks and agents are not affected. - Previous-release saved state: no persisted shape changes. The gate lives in process memory, and runs saved by the previous release are claimed by the same rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ebaffddb01 |
fix(issues): release the outgoing assignee's never-started run on reassignment (#21030)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A lead agent creates tasks and assigns them to other agents > - A task created without an assignee goes to its creator, with a queued run that has not started yet > - When the lead then reassigns the task, the route stopped only a running run. The queued run kept the task's execution lock > - The new assignee's wake waited behind that lock, so nothing ran until the queue reached and withheld the old run > - This pull request releases the outgoing assignee's never-started run on reassignment and wakes the new assignee at once > - The benefit is that reassignment starts the new owner's work immediately, without a stale lock or a duplicate task ## Linked Issues or Issue Description **What happened?** In the paid chat eval `tasks.attachment-brief`, the CEO created a task with no assignee, so it went to the CEO with a queued run. The CEO then reassigned it to Morgan. The CEO's queued run kept the execution lock. Morgan's assignment wake was deferred behind it. After 86 seconds the queue cancelled the stale run as `issue_assignee_changed`. Before that, the CEO got a 403 when it tried to cancel the run, and it created a second copy of the task. **Steps to reproduce** 1. Assign a task to an agent that is busy, so the task's assignment run stays queued. 2. Reassign the task to another agent. 3. The queued run keeps the execution lock, and the new assignee's wake is deferred. **Expected behavior** Reassignment cancels the outgoing assignee's never-started run for that task and wakes the new assignee at once. ## What Changed - `server/src/routes/issues.ts`: on reassignment, when no running run holds the task, also stop the outgoing assignee's queued run that holds the task's execution lock. The stop uses the existing `issue_reassigned` code and the existing stop settlement. - Reassignment stops pass `suppressImmediateRecovery: true`. Without it, releasing the lock queued an `issue_assignment_recovery` run for the outgoing assignee, which took the lock back before the reassignment was saved. The new assignee's wake is the successor. - `server/src/__tests__/issue-reassignment-queued-run.test.ts`: a database-backed route test. It is red on master (the queued run stays queued and holds the lock, and the new assignee's wake is `deferred_issue_execution`). It is green here (the run is cancelled as `issue_reassigned`, and the new assignee has an admitted run for the task). A control case shows that a queued lock holder that is not the outgoing assignee's run is not touched. ## Verification - `cd server && npx vitest run src/__tests__/issue-reassignment-queued-run.test.ts` (2 passed; red on master) - `cd server && npx vitest run src/__tests__/issue-agent-mutation-ownership-routes.test.ts src/__tests__/issue-queued-comments-routes.test.ts src/__tests__/human-directed-work.test.ts` (378 passed) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - Behavior change: a reassignment now cancels the outgoing assignee's queued run that holds the task. Before, that run was cancelled later at claim time with `issue_assignee_changed`. The eval and the run log already recognize both codes. - Reassignment stops no longer queue an immediate recovery run for the outgoing assignee. If the reassignment write then fails, the periodic recovery sweep still recovers the task. - No schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a372271a40 |
fix(wake-queue): keep a task done for its delegator's comment posted during the closing run (#21032)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A lead agent delegates a task, and the assignee works on it in a run > - When someone comments while that run is active, the comment waits as a deferred wake for the next run > - If the run marks the task done, an agent comment with resume intent reopened the task. This started another assignee run and more lead wakes > - When the comment came from the lead that delegated the task, during the run that closed it, the lead gets the completed result anyway > - This pull request keeps the task done in that case. Other agents' feedback and any person's message still reopen it > - The benefit is no repeated work after a task is finished, and no lost person messages ## Linked Issues or Issue Description **What happened?** In the paid chat eval `secrets.least-scope-delivery`, the CEO created Morgan's task. While Morgan's run was active, the CEO posted a redundant instruction comment on it. Morgan's run completed the task. The deferred agent comment then reopened the done task. This caused a second Morgan run and extra CEO wakes, and the eval's run bound was exceeded. #20996 covers a person's owed answers. This change covers agent comments with resume intent. **Steps to reproduce** 1. Let agent A create a task assigned to agent B. 2. While B's run is active, let A comment on the task with resume intent. 3. Let B's run mark the task done. 4. The deferred comment reopens the task, and B runs again. **Expected behavior** The task stays done. A receives the completed result and can follow up. A person's message still reopens the task. ## What Changed - `server/src/modules/wake-queue/application/use-cases.ts`: the "agent feedback after completion" reopen rule now also requires that the comments are not watcher comments posted during the finishing run. Person messages, `issue_reopened_via_comment`, and other agents' feedback do not change. - `server/src/modules/wake-queue/application/ports.ts` and `adapters/postgres.ts`: `isWatcherCommentDuringFinishingRun` is true only when every comment exists, has an agent author and no person author, comes from the task's creator agent or its parent's assignee (not the finishing agent), and was created after the finishing run started. - When this is true, the deferred wake is cancelled with the existing reason "Deferred execution wake no longer applies to a terminal task". The comment stays on the task. - `server/src/__tests__/heartbeat-comment-wake-batching.test.ts`: a new heartbeat case. It is red on master (the task reopens and the assignee runs again) and green here (the task stays done, one run). The existing case, where a non-delegating agent's feedback reopens the task, still passes. - `use-cases.test.ts`: a unit scenario for the new rule. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-comment-wake-batching.test.ts src/modules/wake-queue` (369 passed; the new case is red on master) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - Behavior change: if a delegating agent asks for new work in a comment during the closing run, the assignee does not get it automatically. The delegator receives the completed task and can comment again or create follow-up work. A person's message is never dropped by this rule. - Comments with no author type (older rows) count as agent comments only when they have an agent author and no person author. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d68088465c | Reuse retained Runner descriptions during startup (#21004) | ||
|
|
156453fe7c |
fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A deploy replaces the server. The old server stops its running legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server finds runs whose owner died without that drain (`process_lost`, "server may have restarted"). > - Each of these runs got its follow-up from the bounded transient retry. That retry spends the failure budget (2 attempts) and waits 30 to 60 seconds. > - So the first deploy spent one failure retry. A run that was already the last automatic retry got no follow-up at all. The task then rested on a manual Retry. > - A deploy is routine. An interruption by the server is not a failed attempt, and it must never leave a task waiting for a person. > - This pull request moves these runs to their own `server_restart` retry lane. The lane never spends the failure budget, starts the first continuation at once, and has its own bound. > - The benefit is that a task continues by itself through several deploys a day, and a real loop still stops and shows why. ## Linked Issues or Issue Description **What happened?** A zero-downtime deploy replaced a hosted server. In the 11 minutes around the swap, 11 legacy runs ended with `server_shutdown_interrupted` and 7 ended with `process_lost`. At least 7 tasks were left with no follow-up run and no scheduled retry, so each needed a manual Retry. Two of the stranded runs were already automatic retries. **Expected behavior** Each interrupted run continues once, automatically, on the surviving server, with its original wake. **Paperclip version or commit** Core commit |
||
|
|
8c761abae3 |
fix(environments): wait for a re-activating sandbox provider before refusing a lease resume (#21020)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, a run whose remote Runner kept running
reattaches. First it resumes its exact sandbox lease through the sandbox
provider plugin (`resumeRunLease`).
> - #20982 made a plugin whose worker `initialize` timed out retry its
activation, so the plugin comes back seconds later instead of staying
`error`.
> - Chaos cell CR6 (#20983) shows the gap that remains. The reattaching
run asks for its lease during those seconds and gets
`PluginWorkerUnavailableError`. The setup retry covers that error only
for legacy runs, so the native restart-recovery run fails and needs
Retry.
> - This pull request makes the exact-lease resume wait, for a bounded
time, for the worker of a plugin that is `ready` (re-activating) or
recovering from activation timeouts.
> - The benefit is that a restarted run rides through a short plugin
re-activation instead of failing into Retry.
## Linked Issues or Issue Description
Refs #20982, #20983
**What happened?**
CR6 after #20982 (`w-20261112T041333-s9531`) passed in part. The plugin
re-activated 5 s after its `initialize` timeout, but both tasks finished
with `R3:retry_needed`. Run `a6b3f0f6` resumed its lease at 22:15:35,
inside that window. It failed with `setup_failed` /
`plugin_worker_unavailable` and then `agent_stop_unconfirmed`, because
`run-completion.ts` retries `PluginWorkerUnavailableError` only for
legacy runs (`runtimeMode === "legacy" && nativeSessionId === null &&
runnerInstanceId === null`).
**Expected behavior**
The resume waits for the plugin's worker while the plugin re-activates.
It resumes the same lease when the worker comes back. It still refuses,
as `PluginWorkerUnavailableError`, after the bound.
**Steps to reproduce**
1. Run chaos cell CR6: a K1 restart, then a stall of the replacement's
plugin worker during activation.
2. The reattaching run fails and needs Retry, although the plugin
recovers within seconds.
**Paperclip version or commit**
master at
|
||
|
|
b033cc124d |
fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation catalog checks agent replies and then the human step that follows, such as a person who reveals a key or approves a device > - Some human steps run in a separate private browser process, because they handle credentials. CI keeps the output of that process private on purpose > - Four cases failed only in that private step, so CI showed one fixed message and no cause > - I ran each private step locally against a dev server, where its output is visible, and found the cause of three failures > - This pull request fixes the harness and two product defects, and adds a closed failure record so that the next CI failure names its step > - The benefit is that these human steps work, and a private-step failure is diagnosable without any credential in logs ## Linked Issues or Issue Description Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`). **What happened?** - `routines.webhook-trigger`: the private person opens the routine overview. The trigger cards are only in the Triggers section, so the step waits for a missing button. An unhandled response wait then stops the process. - `connection-setup.secure-api-key`: a key-protected MCP server answers 401 without a challenge header. Paperclip shows "Remote app returned HTTP 401" and does not ask for a key. The person cannot enter the key in the normal flow. - `assistant-connections.device-login`: an expired device code page shows "This item doesn't exist or was moved." The server sends "This code is expired, already decided, or invalid." in the OAuth `error_description` field, and the UI drops it. - `onboarding.new-company`: the private step reads back all saved state and then fails one proof check. CI does not say which check. **Expected behavior** - The person reveals the webhook key in the Triggers section and sends authenticated events. - A pasted MCP link that answers 401 shows the key field. - An expired device code says that the code is expired. - A private-step failure names its fixed step and error class, with no value. **Steps to reproduce** 1. Start a local dev server. 2. Create a routine with a bearer webhook, and run the private webhook worker. It times out on the "Edit webhook" button. 3. Paste the address of an MCP server that answers a plain 401 into Connectors > Add by link. No key field appears. 4. Open `/mcp-device?user_code=<expired code>`. The page shows generic not-found text. **Paperclip version or commit** `master` (reproduced on origin/master before this change) **Deployment mode** Local dev server (embedded PGlite) and the CI chat-eval runner **Installation method** Source checkout (`pnpm dev`) ## What Changed - `integrator-webhook-private-worker.ts`: open `<routine>/triggers` (also for the setup-resume link), and attach a handler to the early response wait so a failed click reports its real step. - `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate` header, to a connection that sends no credential, is handled like a challenge without discoverable sign-in (`oauth_challenge`). The existing UI guidance then asks for a key. A 401 to a connection that already sends a credential keeps the old error. - `ui/src/api/errors.ts`: read the OAuth `error_description` field as a readable server message. - New `private-child-failure.ts`: a private child records only a fixed step name and an error class from a fixed list. The parent echoes a step only when it is in the caller's closed list. Error text, stdout and stderr stay private. - The webhook, private browser and identity children use it. Each has a closed step list file. - `identity-new-company-native-proof.ts`: each proof check has a fixed reason name. The private child reports `new-company-proof-<reason>` when a check fails. The accepted conditions do not change. - Tests: server red/green test for the bare 401, UI red/green test for `error_description`, unit tests for the failure record (no text leaks, closed list only, extra fields dropped). ## Verification - Local dev server (embedded PostgreSQL), private workers run directly with the real child process: - Webhook, ready and setup-pending triggers: before, timeout on "Edit webhook". After, receipt with invalid status 401, valid status 202 and a linked task. - Secure key entry with the synthetic notes provider: before, timeout on the key field. After, an active connection with health `ok` and 2 authorized `tools/list` calls. - Expired device page: before, `at step device-expired-page (TimeoutError)`. After, `expiredPageVisible: true`. - `server`: `vitest run src/__tests__/tool-access-service.test.ts -t "pasted link answers"` (fails before the discovery change with 502, passes after). - `ui`: `vitest run src/api/errors.test.ts` (fails before, passes after). - `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server `tsc --noEmit`. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: unchanged. - One paid run of the four cases on this branch. Results are in a PR comment. ## Risks - A plain 401 from an MCP link now returns 422 `oauth_challenge` instead of 502. The draft connection is removed, as for other challenges. Agents that call the connect API see the "needs sign-in" answer instead of a raw status. This only applies when the connection sends no credential. - Some API errors that carry `error_description` now show that text instead of generic copy. That text is written for people. - The failure record is opt-in per child and contains only constants. The parent ignores a record with any other field. - `onboarding.new-company` is not fixed here. Its native hello probe cannot start locally on macOS, so the failing check is not yet known. This change makes the next run name it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local browser automation through Playwright). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1bb4dcbe4b |
fix(routines): agents can only create authenticated webhook triggers (#21006)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routines can have webhook triggers that outside services call > - Agents can create and change routine triggers through the API > - The choice of webhook authentication belongs to a person > - This pull request lets agents create authenticated webhook triggers only, and keeps every choice open to people on the board > - The benefit is that agents can only create authenticated webhook triggers ## Linked Issues or Issue Description **What happened** In a chat eval, an agent set up a routine webhook trigger with a weaker authentication choice than the default. **Expected behavior** Agents can only create authenticated webhook triggers. When a sender cannot authenticate, the agent asks a person, who decides on the board. **Steps to reproduce** 1. Ask an agent in chat to give a routine a webhook for a test service. 2. Read the trigger that it creates. ## What Changed - `server/src/services/routines.ts`: trigger creation, trigger updates and revision restores refuse an agent that does not keep the webhook trigger authenticated. The error tells the agent to use the default or a signed mode, or to ask a person. Board users keep every option. An agent can still edit other fields of a trigger that a person set up. - `skills/paperclip/references/routines.md`: says agents can only create authenticated webhook triggers. - Test: new case in `routines-service.test.ts`. ## Verification - `cd server && npx vitest run src/__tests__/routines-service.test.ts`: 104 passed. The new test fails without the service change (red) and passes with it (green). - `npx vitest run src/__tests__/routines-routes.test.ts src/__tests__/company-portability.test.ts`: passed. - The chat eval `routines.webhook-trigger` already requires one authenticated trigger. It runs again on a combined throwaway branch; results are in a PR comment. ## Risks - An agent-run company import that would add such a trigger now fails with the same error and rolls back. A person can run that import from the board. - Agents must use the default or a signed mode. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5a7378af56 |
fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts queued runs. The run-dispatch worker re-runs the queue only when a write sends a "queued run dispatch" signal. > - A follow-up run can queue while its predecessor still holds its sandbox lease. The claim then declines the run without a log line, and the run stays `queued`. > - After a controller restart, nothing sent a new signal once that lease was settled. The run stayed `queued` forever. The issue had no Retry, and the reconciler counted the queued run as a valid next action. > - This pull request sends a dispatch signal from the native finalization writes that can clear a predecessor's cleanup gate, logs why a claim was declined, and bounds the wait. The dispatch worker re-drives an undispatched queued run after 60 s as a backstop, and the ownership reconciler reports and re-drives one still queued after 5 minutes. > - The benefit is that a follow-up starts within seconds of its predecessor settling, a lost signal delays it by about a minute instead of forever, and a run that still cannot start is visible. ## Linked Issues or Issue Description Refs #20909, #20935, #20953, #20970 **What happened?** Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` + #20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold, `--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the outage with an accepted result. The next controller finalized it at 22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z (`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source `native_status_decision`). The run stayed `queued` with `claimed_at = null` until the wave ended 19 minutes later. The agent stayed `running` with no running run. The task had no Retry. The grade was `R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`. **Root cause** `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`, `lockIssueExecutionClaim`) declines a queued run while `hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is true for its predecessor. After the restart, the predecessor's warm `reuse_by_environment` lease was still `active` when the continuation queued, so the first claim was declined. The decline is correct, but nothing owned the retry. `registerRunDispatchWork` (`server/src/services/heartbeat/dispatch-work.ts`) has `next: async () => null`: the queue is only re-run by a later dispatch signal. At wave end the lease was `retained` with `cleanupStatus: success`, which does not block a claim. A dispatch pass in that state starts the run (verified locally). So no pass ran after the gate cleared. The ownership reconciler listed the run as `queued_run` with owner `run_dispatch` (always "alive"), and I7 treats any queued run as a next action, so nothing reported it. This is not the #20935 throttle. The wave base predates #20935, and a throttled intent keeps its wake intent `queued` with a `nextAllowedAt` deadline. Here the run row existed and its claim was never retried. On current master #20935 parks this specific `response_wake` self-wake, but any continuation, comment wake or Retry that queues behind a settling predecessor after a restart has the same gap. The red test uses a `same_agent` continuation, which master still enqueues. **Steps to reproduce** `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart". ## What Changed - Event-driven dispatch. A lease release already signals queued-run dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`, stop-and-retain settlement). I enumerated every `environment_leases` status writer: each one that moves a lease out of a blocking state already signals. The writers that do not signal only acquire a lease or set `pendingCleanupInFlight`, so they add a block and do not clear one. The native coordinator side had no signal, so: - `notifyNativeFinalizationWork` (the shared notifier of coordinator commits and terminal transitions) now also signals queued-run dispatch for the company. - The coordinator writes that clear `leaseOwner` or end the phase outside that notifier now signal in the same transaction: `releaseLapsedCoordinatorOwner`, native startup cleanup, observed-workspace preservation, `record_finalization_error` in the interaction bridge, the wake-queue terminal recovery, and the reconciler's ambiguous-state terminal failure. - Not changed: three non-transactional pre-dispatch closes in `run-completion.ts` and `agent-execution-stop.ts`. Their callers queue or settle the successor after the write, so the successor's own enqueue starts the queue. - `claimQueuedRun` now logs the reason when it declines a queued run at the issue execution claim (`predecessor_cleanup_pending`, `predecessor_still_executing`, `issue_execution_owned_by_another_run`, case handoff, task chat cleanup), when a predecessor is still settling in this process, and when a temporary budget hold keeps it queued. - New `server/src/services/queued-run-dispatch-liveness.ts`: `listUndispatchedQueuedRuns` reads queued runs in active companies whose ready, non-paused agent has no running run. It reads every page with a keyset cursor until the set is exhausted, so older held runs cannot hide a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each such run 60 s after it was created, then every 2, 4, 8 minutes, capped at 15 minutes. Run ids that leave the set are dropped. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch worker's `next` now plans those re-drives. It re-reads only the companies the pass just dispatched, keeps the deadlines of every other company, and runs no query when the pass dispatched nothing. Only an instance-wide pass (startup or explicit wake) reads every company. It adds each due company to the dispatch scopes and returns the next due time. - `packages/db`: new partial index `heartbeat_runs_queued_created_idx (company_id, created_at, id) WHERE status = 'queued'` (migration `0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in `client.ts`). It returns `null` (no deadline, no SQL until a signal) when no run is undispatched or scheduling is suppressed. Every re-drive goes through the normal `startNextQueuedRunForAgent` and `claimQueuedRun` gates, so it never bypasses a hold, budget, pause or cleanup gate. - `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns` into the worker (two lines). - `server/src/services/recovery/ownership-invariant-report.ts`: new check. A queued run still undispatched 5 minutes after creation is an I7 violation (`kind: queued_run`, `reason: queued_run_not_dispatched`), written to the activity log once. A younger one sets the report's next due time. - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `applyOwnershipReportBackstops` sends a queued-run dispatch signal once, for each company with a newly found violation. Repeat passes do not signal again, so a held run does not wake the report every 30 s. - `tests/restart-chaos/README.md`: records the wave and the fix in the CR results table. The existing CR1 cell (#20909) and the enforced U1 check already cover this class in the harness. ## Verification - RED on master (`23909077ad`): - `heartbeat-native-restart-suspension.test.ts` "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart": the continuation is `queued` behind the predecessor's `active` warm lease. The run-dispatch worker is registered, the lease then settles to `retained`/`success` with no dispatch signal, and the run stays queued: `AssertionError: expected null not to be null` after 75 s. - `ownership-reconciliation-report.test.ts` "reports and re-drives a queued run that no dispatch started while its agent had nothing running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no violation for a run queued 19 minutes). - `ownership-reconciliation-report.test.ts` "signals queued-run dispatch when finalization work settles or a finished coordinator's lapsed owner is released": `expected [] to include 'settled'`. - GREEN on this branch: all pass. The restart test has two cases. When the lease is released through `releaseLease`, the continuation starts within 5 s. When the lease is settled by a write with no signal, the backstop starts it within 75 s. A new report test puts three older held runs ahead of a stranded run with a page size of 2, finds the stranded run, and checks that the backstop signals once. - `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap, and going quiet. - Focused suites pass (22 files, 495 tests): recovery-deadline-acceptance, live-recovery-work, heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work, deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the ownership kinds suites, and others. `heartbeat-native-restart-suspension` and `ownership-reconciliation-report` pass in full. `tsc --noEmit` for the server passes. ## Risks - A queued run that is held on purpose (for example a temporary budget hold or a case handoff waiting for release) is now re-checked at most every 15 minutes after the first minute. Each re-check is one indexed query and the normal claim gates. Such a run is also reported once as an I7 violation. - An instance with an undispatched queued run keeps a dispatch deadline, so it does not idle while that run waits. An instance with no such run is unchanged. - Schema: one additive partial index, built concurrently. No data change. State saved by an older version is read as is. Nothing runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
82e0393598 |
fix(ui-assets): retry a throttled asset upload and never fail startup on a storage outage (#21010)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted Paperclip server can run volume-free. Before it serves, it publishes its UI build assets to shared object storage, so an older browser tab can still load its chunks after the old container is gone > - A build has several hundred assets. The server uploads them one after another at startup > - Object storage throttles a burst with `SlowDown` (HTTP 503). The upload body is a stream, so the storage client cannot replay the request, and the error leaves `startServer` > - The process exits, the host restarts it, and the next start uploads the same assets again. A workspace stayed down while four starts in a row failed this way > - This pull request retries a throttled asset and lets the server start when storage stays unavailable > - The benefit is that a storage slowdown cannot keep a server from starting ## Linked Issues or Issue Description Refs #20990, #20952 (other startup failures found on the same hosted workspace). **What happened?** A hosted volume-free workspace received a new release. Each start logged many `An error was encountered in a non-retryable streaming request.` lines and then `Paperclip server failed to start` with `SlowDown: Please reduce your request rate.` for a key under `ui-assets/v1/`. The four starts failed after 44, 42, 79 and 90 seconds, each on a later asset name. **Expected behavior** The server waits and repeats a throttled upload. If shared storage stays unavailable, the server starts and serves its build from local files. **Steps to reproduce** 1. Run a server with `PAPERCLIP_SHARED_UI_ASSETS=true` and S3-compatible storage. 2. Make the storage answer `SlowDown` (HTTP 503) to one `PutObject` during startup. 3. The server exits with `Paperclip server failed to start`. **Paperclip version or commit** `52cdf76bf3e0cdf29037b63bf0f5e99aa464eab7` ## What Changed - `retainedUiAssets.publish`: each asset is published in a retry loop. A throttled or server-side storage failure waits (0.5 s to 8 s, seven waits at most) and repeats the asset with a fresh stream. - `isTransientStorageError`: recognizes `SlowDown`, throttling, request timeout and HTTP 429/500/502/503/504 answers. - `app.ts`: when publication still fails with a transient storage error, the server logs an error and continues. Other failures, such as a name collision with different bytes, are still fatal. - Tests for the retry, for the give-up path, and for the error classification. ## Verification - `npx vitest run src/__tests__/retained-ui-assets.test.ts` in `server/`: 4 tests passed. The two new tests fail on `master` because `publish` rejects on the first `SlowDown` and takes no `sleep` option. - Not run locally: the full suite. CI supplies that coverage. - Not verified: a live start against throttling storage. ## Risks - A start can take up to about 31 seconds longer for each throttled asset before it gives up. - If storage stays unavailable, the server starts without its assets in shared storage. An older browser tab can then fail to load a chunk after the old container is gone. The server logs this at error level. - Collision detection is unchanged. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
a9a08d8169 |
fix(plugins): retry a plugin activation whose worker initialize timed out (#20982)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins, sandbox providers such as Daytona among them, run as worker
processes that the server activates at boot. Activation sends the worker
an `initialize` RPC with a 15-second timeout.
> - When `initialize` timed out, the loader marked the plugin `error`
for the life of the server process. Only the next boot or an operator
retried it.
> - For a sandbox provider, that means every Daytona run on that server
fails. New runs fail with `lease_acquire_failed … plugin is currently
error`. Reattachments fail with "The exact sandbox lease cannot be
resumed by its verified provider". Retry hits the same dead plugin.
> - A worker that is slow or stalled for a moment at boot is transient,
not a broken plugin.
> - This pull request retries an activation whose `initialize` timed
out, with a bounded backoff. Meanwhile the server keeps serving, and
runs that need the plugin see a restarting worker, which setup retries.
Deterministic failures stay a permanent error.
> - The benefit is that a short plugin stall at boot no longer disables
a provider until the next deploy.
## Linked Issues or Issue Description
Refs #20966, #20857
**What happened?**
Chaos wave `w-20261111T232850-s9521` (cell CR4, first version) stopped
the replacement server's Daytona plugin worker while the server was
activating it. The worker's `initialize` RPC timed out after 15 s
(`INITIALIZE_TIMEOUT_MS`). `plugin-loader.ts` then called
`lifecycleManager.markError`, and the plugin stayed `error`. Task t03's
reattachment failed in `resumeRunLease`, and its Retry did not start a
new run. Task t04 failed 18 s after launch with `lease_acquire_failed`
("…plugin is currently error"), before its own fault.
**Expected behavior**
An activation timeout is retried, and the plugin ends `ready` when the
worker answers. Runs that need the plugin meanwhile are retried within
their bounded setup budget. A bad manifest, a worker that exits during
`initialize`, or a version mismatch is still a permanent error.
**Steps to reproduce**
1. Start the server.
2. Stop (SIGSTOP) a plugin worker for more than 15 s while the server
activates it.
3. Resume the worker. The plugin stays `error` until the next boot.
**Paperclip version or commit**
master at
|
||
|
|
7d9973405c |
fix(chat): ask for a missing name only when the request and saved brief give none (#21007)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to their agents in chat, and the chat guidance tells the agent how Paperclip works > - #20888 added a rule: ask for a missing name before building something that must carry it > - In a chat eval, "Start a Launch project and have Morgan build its landing page" made the agent create the project and then ask for the brand name. It did not create Morgan's task > - The person had named the project, so a question was not necessary > - This pull request narrows the rule: ask only when the request, the conversation and the work's saved brief give no name > - The benefit is one necessary question when a name is truly missing, and no extra question when the person already named the work ## Linked Issues or Issue Description Refs #20888 **What happened** For "Start a Launch project and have Morgan build its landing page", the agent created the Launch project and asked for the brand name. It did not create the landing-page task for Morgan. **Expected behavior** The agent uses the name the person gave (Launch), creates the project and the task for Morgan, and returns both links. For "Build a landing page for my new product." with no name anywhere, the agent still asks one free-text question for the product name. **Steps to reproduce** 1. Run the chat eval `tasks.website-without-project` on master. 2. Read the reply: the agent asks for a brand name and creates no task. ## What Changed - Chat guidance (`AGENT_CHAT_DIRECTIVE`): use the name that the request, the conversation or the work's saved brief gives. A project the person names in the request counts. Ask only when none of these gives a name. Do not take the name from unrelated records such as other existing projects or the company name. The rule against invented or placeholder names stays. - Tests: updated and new text checks in `agent-conversations.test.ts`. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts`: 64 passed. The new test fails on master (red) and passes with this change (green). - `npx vitest run` for `board-access-denial` and `codex-local-execute`: passed. - Paid chat evals `tasks.website-without-project` and `questions-and-approvals.necessary-question` run on a combined throwaway branch; results are in a PR comment. The necessary-question fixture has an unrelated existing "Launch" project, so the guidance says not to take the name from other projects. ## Risks - Guidance text only. The model can still ask or not ask in edge cases. The paid evals check both directions. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
601babe322 |
fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages
## Linked Issues or Issue Description
**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.
**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.
**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.
**Paperclip version or commit**
master at
|
||
|
|
2a1620f143 |
fix(heartbeat): deliver a person's answer that waited behind a turn that closed the task (#20996)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question on its task, and the person
answers it in the task chat
> - The person can answer while the agent's turn is still running. The
answer then waits for that turn to finish
> - If that turn completes the task, the release step cancels every
waiting wake for the assignee on the done task. The person's own queued
messages reopen the task, but the answer was dropped and no turn
received it
> - A person who answers an agent's question expects the agent to get
the answer. Losing it without a trace is a silent failure
> - This pull request treats an answer saved before the task was
completed as owed input. It reopens the task and delivers the answer
once
> - The benefit is that a person's answer is never dropped because the
agent finished its turn first
## Linked Issues or Issue Description
**What happened?**
The paid chat eval `steering.approval-delivery` showed it. The person
answered the agent's naming question while the agent's turn was running.
The turn completed the task. The answer's own wake was then cancelled
with "Deferred execution wake no longer applies to a terminal task". No
turn received the answer.
**Steps to reproduce**
1. Start a turn for the assignee on a task.
2. While the turn runs, answer the agent's question on that task.
3. Let the turn mark the task done.
4. The answer's wake is cancelled, and no run receives it.
**Expected behavior**
The answer reaches the assignee exactly once. An answer saved after the
task was completed is still dropped, as before.
Refs #20981, #19331
## What Changed
- `wake-queue/application/use-cases.ts`: before cancelling a deferred
assignee wake on a done task, check whether it carries a person's owed
answer. If it does, reopen the task to `in_progress` and promote the
wake. An answer continues the assignee's own execution, so the existing
continuation status gate applies unchanged.
- `wake-queue/adapters/postgres.ts`: add `owedInteractionAnswer`. It
applies only to an undelivered response that a person resolved, that is
not a tool action, and that was saved at or before the task's
`completedAt`. It returns `blocked` when unresolved blockers forbid
`in_progress`. `reopenIssue` takes an optional target status.
- The blocked case cancels the answer wake with an explicit reason and
does not reopen the task, so the release step does not fail. A
regression test reproduces the earlier throw (fix for the Greptile P1).
- `heartbeat-answer-behind-settling-run.test.ts`: an end-to-end
heartbeat test. It is red on master (the answer wake is cancelled and no
run is bound to it) and green here (one run is bound and succeeds). A
control test shows that an answer saved after completion is still
dropped.
- `use-cases.test.ts`: unit cases for an owed answer, a not-owed answer,
a cancelled task, and an agent-requested wake.
- Eval `steering.approval-delivery` oracle v7: also accept the Acorn
document when the worker's settling turn saved it after the answer was
saved. The eval still requires exactly one later turn bound to the
answer. Calibration tests cover the retained shape, reworded text, a
pre-answer save, the wrong name, another agent, a stale revision, and a
missing bound turn.
## Verification
- `cd server && npx vitest run
src/__tests__/heartbeat-answer-behind-settling-run.test.ts
src/modules/wake-queue
src/__tests__/heartbeat-blocked-question-answer.test.ts` (351 passed)
- Red check: with the `server/src/modules` changes reverted, the new
delivery test fails.
- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/question-response-delivery.test.ts` passed. In
`issue-queued-comments-routes.test.ts` all 138 tests passed, but a
cleanup hook timed out locally. CI covers this file.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts` (58 passed)
- `tsc --noEmit` passes for `server` and `tests/runner-e2e`.
- Paid single-case run 43136466619 on
|
||
|
|
faa78dd818 |
fix(agents): keep a managed instructions bundle managed when a restarted volume-free server saves another setting (#20997)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent settings are saved through `PATCH /api/agents/:id`. On a
volume-free server, a save is refused when the agent's instructions are
external, because files outside the managed root do not survive a
restart
> - A managed bundle saves its instruction paths as absolute paths under
the home of the server that wrote them. A replacement volume-free server
can start with a new home
> - After such a restart, any `adapterConfig` change re-derived the
bundle mode from the saved `instructionsFilePath`. That path is outside
the new managed root, so the bundle was read as `external` and the save
failed with 422 "Volume-free servers require managed agent instructions"
> - This pull request keeps a managed bundle managed when the save keeps
its saved `instructionsFilePath`, and rewrites its paths under the
current managed root
> - The benefit is that a board user can change one setting (for example
`idleTimeoutMs`) after a restart, and a real switch to external
instructions is still refused
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave `w-20261112T012841-s7000` (volume-free mode,
Daytona, Codex, restarts K1 and K15 with a fresh server home), the board
sent `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`. All 3 tasks of the W2C
workload got 422 "Volume-free servers require managed agent
instructions".
**Expected behavior**
A partial `adapterConfig` change of an agent with managed instructions
succeeds on a restarted volume-free server.
**Steps to reproduce**
1. Create an agent with a managed instructions bundle on a volume-free
server.
2. Restart the server with a new `PAPERCLIP_HOME`.
3. Send `PATCH /api/agents/:id` with
`{"adapterConfig":{"idleTimeoutMs":301001}}`.
4. The server returns 422.
**Why Codex looked different from Claude**
It is not the adapter. The Claude wave that passed
(`w-20261112T010036-s6801`) had no faults, so its PATCH ran on the
server that wrote the bundle. The Codex wave sent its PATCH after K1 and
K15 restarts with a new home. The new test fails the same way for a
Claude agent after a fresh-home restart, and passes for both adapters on
the original server.
**The PATCH contract**
`adapterConfig` merges into the saved config unless the request sets
`replaceAdapterConfig: true` (validator in
`packages/shared/src/validators/agent.ts`, route in
`server/src/routes/agents.ts`). The UI full-config save sends
`replaceAdapterConfig: true` with the whole config. The merge was
already correct. The fault was in
`syncInstructionsBundleConfigFromFilePath`, which the route applies to
the effective (merged) config: it classified the bundle only from the
absolute file path and the current managed root. The contract does not
change.
## What Changed
- `server/src/services/agent-instructions.ts`
(`syncInstructionsBundleConfigFromFilePath`): when the saved bundle is
managed and the effective config keeps the saved `instructionsFilePath`,
keep it managed with the root that `deriveBundleState` already resolves
(the current server's managed root) and the saved entry file. A
different `instructionsFilePath` is classified as before.
- `server/src/__tests__/agent-volume-free-config-patch.test.ts`: route
tests with embedded Postgres for a Codex agent and a Claude agent: a
one-setting change after a fresh-home restart, the same change on the
original server, and a real switch to an external path, which is still
refused.
- `doc/DEVELOPING.md`: describe how a volume-free server keeps a managed
bundle across a new home.
## Verification
- Red on `origin/master` (`9dfdeaad86`), `cd server && npx vitest run
src/__tests__/agent-volume-free-config-patch.test.ts`:
- `× changes one codex setting after a fresh-home restart and keeps
managed instructions`
- `× changes one claude setting after a fresh-home restart and keeps
managed instructions`
- `AssertionError: {"error":"Volume-free servers require managed agent
instructions"}: expected 422 to be 200`
- The 4 other cases passed (the original server, and the refused
external switch).
- Green with the fix: the same file passes 6/6, together with
`agent-instructions-service`, `agent-instructions-routes`,
`ephemeral-server-contract` and `agent-configuration-save` (65 tests).
- The restart-chaos W2C workload (`tests/restart-chaos/rewake.ts`)
already sends this PATCH after K1/K15 restarts and caught the fault as
`T:config_changed_before_wake`. No new chaos cell is necessary.
## Risks
- Low risk. The change applies only when the saved bundle is managed and
the save keeps the saved `instructionsFilePath`. Then the paths are
rewritten under the current managed root, which is the root that bundle
reads already use.
- The volume-free guard is unchanged. A save that sets an
`instructionsFilePath` outside the managed root is still refused (tested
for both adapters).
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
1949ac6788 |
test(chaos): fail the directory-lock crash point with the lock's own busy error (#20988)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness can fail one recovery step on purpose (`--fail-recovery-step`), through a test-only crash point in the server > - The `directory-lock` step fails `agent_directory.operation_lock.before_acquire`, the per-agent Postgres lock that a volume-free server takes around remote agent-directory work > - The real lock path throws `AgentDirectoryOperationBusyError` when another run holds the lock. Preparation turns that error into a deferral and retries later > - The crash point threw a plain 409 `conflict()` with the same message, so the deferral never matched and the injected fault failed the run. A volume-free wave on 2026-11-12 showed this: the run failed with `native_continuation_requires_reconciliation` and needed Retry, where real contention would have been deferred > - This pull request makes the crash point throw the lock's own error class > - The benefit is that the `directory-lock` chaos cell tests the product's real handling of a busy agent directory ## Linked Issues or Issue Description **What is the problem?** The test-only crash point `agent_directory.operation_lock.before_acquire` throws a different error class from the lock it stands in for. **What is the expected behavior?** A run that meets the injected failure takes the same path as a run that meets real lock contention. **Steps to reproduce** Run a volume-free Daytona wave with `--fail-recovery-step directory-lock:1`, workload W2 and a K2 restart. The recovering run fails with `native_continuation_requires_reconciliation`; the stack shows the crash point inside `prepareCopy`, under `prepareWithAgentDirectoryDeferral`, which only defers `AgentDirectoryOperationBusyError`. ## What Changed - New test `server/src/__tests__/agent-directory-lock-crash-point.test.ts`. - `server/src/services/agent-directory-working-copies.ts`: the crash point's `fail` returns `new AgentDirectoryOperationBusyError()` (same 409 status and message) instead of a plain `conflict()`. ## Verification - `server/src/__tests__/agent-directory-lock-crash-point.test.ts` arms the point in fail mode through a control file (as restart chaos does) and checks that the injected failure is an `AgentDirectoryOperationBusyError` that `prepareWithAgentDirectoryDeferral` turns into a deferral. It fails on master (the injected error is a plain `HttpError`) and passes with this change. - Server typecheck passes. - The crash point is disarmed unless a chaos wave arms it, so production behavior does not change. - Live check: the next volume-free `directory-lock:1` chaos wave (W2, K2, Codex) shows whether the recovering run is deferred instead of failing. I will add the result to this PR. ## Risks - Low risk. The change is in a test-only crash point that is disarmed in production. The README step table already documents the error as the 409 "Agent directory operation is already active". ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
52cdf76bf3 |
fix(ephemeral): start a volume-free server over harness backups a volume-free server left unsaved (#20990)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted Paperclip server can run volume-free: its durable state is in the database and object storage, and its local disk is replaced with every deployment > - At startup, a volume-free server runs a readiness gate that refuses to start when it finds data that still needs migration off a volume > - One of those findings is a harness backup that never reached durable storage. A volume-free server that stops in the middle of a run always leaves some > - The gate then refuses every later start, the process exits, and the host restarts it into the same refusal. The workspace stays down until a person repairs it > - This pull request lets a server start over such backups when its filesystem carries the volume-free owner marker > - The benefit is that a volume-free server recovers from an interrupted run by itself ## Linked Issues or Issue Description Refs #20952 (the same crash loop for Runner controller files; this is the next check in the same gate). **What happened?** A hosted volume-free workspace stopped while runs were in progress. Every later start logged `Paperclip server failed to start` with `Volume-free server data is not eligible: native_harness_backups_require_migration`, and exited. The host restarted the container and the server exited again. The workspace did not serve any request for more than two hours. **Expected behavior** A volume-free server starts after an interrupted run. Restart recovery handles the runs whose local harness backup was lost with the old container. **Steps to reproduce** 1. Run a volume-free server (`PAPERCLIP_EPHEMERAL_SERVER=true`) with native Runner tasks in progress. 2. Stop the server process before the harness backup of an active session reaches durable storage. 3. Start the server again. It refuses to start with `native_harness_backups_require_migration`. **Paperclip version or commit** `87fe4d979355232570428deba480b47e1812a26c` (contains #20952). ## What Changed - `assertEphemeralServerData`: when the only reason is `native_harness_backups_require_migration` and the filesystem carries the volume-free owner marker (`volumeFreeControllerStateOwned`), the gate removes that blocker, logs a warning, and lets the server start. - A filesystem without the marker is refused as before. All other blockers are unchanged. - `readEphemeralServerDataEligibility` is unchanged, so the migration and enrollment callers still see the blocker. - Added `ephemeral-server-startup-run-state.test.ts`. ## Verification - Red: with the change to `ephemeral-server-contract.ts` reverted, `starts a volume-free server over harness backups a volume-free server left unsaved` fails (1 failed, 2 passed). - Green: `npx vitest run src/__tests__/ephemeral-server-startup-run-state.test.ts src/__tests__/ephemeral-server-data.test.ts` in `server/`: 2 files, 8 tests passed. - Not run locally: the full suite. CI supplies that coverage. - Not verified: a live volume-free start. The hosted workspace that showed the failure will receive this change through the normal release. ## Risks - A server on a marked volume-free filesystem now starts with unsaved harness backups. Those backups were on a disk that no longer exists, so the refusal protected nothing. The affected runs depend on restart recovery. - The marker is the only evidence that there is no volume. A filesystem that still is a volume and does not carry the marker keeps the refusal. - The gate does not repair or delete the stale lease stamps. The warning repeats at each start until restart recovery settles those runs. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5d1bb5192f |
fix(retry): carry the original wake to every re-queued run (#20989)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A run serves one wake: a comment, an answer, an assignment, or a one-shot task monitor check (`issue_monitor_due`). > - When a run fails, Paperclip often starts a successor run by itself: the bounded transient retry, the uncharged `interrupted_preparation` re-queue, workspace or connection waits, missing-comment repair, capacity retries and recovery successors. Each successor links to the failed run through `retry_of_run_id`. > - Most of these re-queues replace the wake reason with their own (for example `transient_failure_retry`). The agent then does not know which wake it must handle. > - #20970 fixed this for the board Retry only. A no-fault chaos wave showed the same loss on an automatic retry: the monitor run failed before provider work, the automatic retry succeeded, and the agent did not treat it as the monitor check. The monitor was already consumed, so the task rested `in_progress` with nothing pending. > - This pull request covers all re-queue paths at one shared point: the continuation builder that every issue dispatch uses (`buildExecutionContinuation`, called from `heartbeat/task-context.ts`). It walks the re-queue chain to the first run that was not a re-queue and records that run's wake reason as `trigger.retriedReason`. The wake prompt already shows that field (#20970). > - The benefit is that every successor of a failed run handles the same wake as the run it replaces. ## Linked Issues or Issue Description **What happened?** No-fault restart-chaos wave on master `87fe4d9793`, volume-free mode (`w-20261112T010036-s6801`, Claude, task t01, workload W2). The monitor wake started run `78e27ef9`. It failed before provider work (`native_predecessor_retirement_unproven`). The `interrupted_preparation` re-queue (attempt 1/12) started run `0cd387a0` on a fresh sandbox, and that run succeeded. The agent reported that no board comment arrived and that the monitor wake was not visible. It yielded `response_wake` again. The monitor was `triggered` with no `nextCheckAt`. The issue rested `in_progress` with `canRetry=false`, no live run, no wake and no monitor. The harness graded `R1 rested_not_done` and `R3 missing_retry`. **Expected behavior** A successor of a failed run tells the agent which wake it re-delivers. A re-queued monitor check is handled as the monitor check. **Steps to reproduce** 1. Wake a native agent with `issue_monitor_due` on a task. 2. Make the run fail before provider work, for example when the previous run's sandbox session cannot be retired. 3. Let the `interrupted_preparation` re-queue run. 4. Before this change, the successor's continuation trigger has the re-queue reason only, and its prompt does not name the monitor check. **Additional context** Follow-up to #20970. Refs #20970 Refs #20935 Refs #20863 ## What Changed - `server/src/services/execution-continuation.ts`: new `requeuedWakeReason`. - It starts at the current run's `retry_of_run_id` column. If that is empty, it uses the context `retryOfRunId`, or the source run of a board Retry. - It follows the chain within the same company and issue, at most 32 links, until it finds a run with no re-queue link. It returns that run's wake reason. - A `trigger.retriedReason` already saved on a run in the chain is reused. - `trigger.retriedReason` is set only when the original reason differs from the current reason and is not itself a Retry reason. A re-queue that keeps the original wake reason (`preserveWakeContext`) is unchanged. - This one point covers every successor that links with `retry_of_run_id`: bounded transient retries, `interrupted_preparation` (#20863), workspace and AI-connection waits, missing-comment repair (`run-lifecycle.ts`), native capacity retries, safe replacement, and the board Retry. - Tests: - `server/src/__tests__/heartbeat-native-predispatch-retry.test.ts`: the assembled heartbeat service runs a monitor-due wake, a predecessor-retirement failure before provider work, and the automatic `interrupted_preparation` re-queue. It asserts the successor's saved continuation and wake prompt. - `server/src/services/execution-continuation.test.ts`: a two-link chain of automatic re-queues, and a re-queue that keeps the original reason. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/heartbeat-native-predispatch-retry.test.ts -t "delivers the monitor check"`. The successor's trigger had `sourceRunId` = the failed monitor run, but no `retriedReason`. - Green: `heartbeat-native-predispatch-retry.test.ts`, `execution-continuation.test.ts` and `heartbeat/retries.test.ts` pass (78 tests). - `npx tsc --noEmit -p server` passes. - Chaos: the existing no-fault W2 cell in volume-free mode reproduces this when the monitor run fails before provider work. I did not rerun a wave. ## Risks - Low risk. The change adds one optional trigger field to more wakes. It changes no admission, status or authority path. - The walk costs at most one indexed primary-key read per chain link, only for runs that are re-queues. - The walk follows only same-company, same-issue runs. Old saved runs work, because they already store `wakeReason` and the `retry_of_run_id` column. - The resting state "agent yielded `response_wake` after the monitor was consumed" is still possible when an agent really waits. #20935 shows it on the run projection as "Waiting for a reply". The ownership reconciler's I7 check still counts it as `no_owner_no_wake`. This PR does not change I7. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8818402e7a |
feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects requests that need a human decision. > - Teams need to know who owns each request and who may act on it. > - A shared close operation must differ from hiding a card for one person. > - This pull request adds personal routing and durable outcomes to the existing attention service. > - People can claim or transfer work and inspect what another person did for them. > - Domain permissions still decide who may resolve each request. ## Linked Issues or Issue Description **Subsystem affected** Server, shared contracts, and database. This is the routing foundation for a stacked inbox UI change. **Problem or motivation** A company-wide attention feed does not give teams a clear personal queue. Archive, withdrawal, transfer, and decisions by another person need distinct scopes and records. **Proposed solution** Resolve a request audience from its addressee, task participants, and configured owners. Provide personal, unassigned, company, all-assignee, and archived views. Add claim, transfer, personal archive and snooze, shared withdrawal, receipts, and personal history. **Alternatives considered** A single shared inbox leaves ownership and personal hiding ambiguous. The service keeps one canonical request and projects its audience instead of copying requests for each person. **Roadmap alignment** This extends the existing Multiple Human Users and Agent Reviews and Approvals capabilities in ROADMAP.md. **Additional context** Refs #19363 and #19520 for the current inbox navigation and confirmation replacement rules. Refs #20268 for related actionable-request work. That PR filters actionability; this stack adds audience routing and personal outcomes. This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this branch. ## What Changed - Add nullable company and project decision owners and a company budget owner. - Add audience views and personal badge counts to the canonical attention projections. - Add claim, transfer, withdrawal, personal archive, snooze, and restore operations with permission checks and activity records. Lock addressed transfers to the current addressee or routing administrators, and validate the recipient against canonical resolution policies. - Add receipts and scoped history for work handled by another person. - Keep closed-task requests and replaced confirmations out of current cards. - Add an idempotent migration and a concurrent receipt-history index. - Update typed fixtures so the existing UI remains compatible with the new contracts. ## Verification - Server layered typechecking passes, including all three test-tree partitions. - Seven routing and interaction-resolution suites pass all 249 tests against a dedicated external PostgreSQL test database on `4caecc21d8`. Six new regressions failed before the repair; they cover takeover followed by approval, legitimate delegation, excluded creators/review requesters, and unauthorized claims. - Prior focused verification covers private approval history and receipts, transfer visibility, audience escalation, withdrawal governance, hidden-card counts, per-recipient confirmation replacement, and SDK source contracts. - A synthetic three-person company passed 37 routing and lifecycle acceptance checks. The UI stack adds browser acceptance for the shared controls. - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331) passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with no open findings. The fresh security scan passes after explicitly rechecking the addressee-transfer repair. - `pnpm verify` reports exit 2 because a CI-global script changed; full CI supplies that broader coverage. Embedded PostgreSQL could not start on this Mac because shared-memory slots were exhausted, so local database checks use a separate external test database. Linux CI covers the embedded setup. ## Risks - Personal badge counts change to the viewer's assigned requests. Accessible feed totals and task membership keep their existing scope. - The migration adds nullable owner fields and a dismissal fingerprint. Its index runs outside a transaction. The migration is classified and rerunnable. - Routing changes the addressee used by resolution authorization. The current addressee or a routing administrator must authorize an addressed transfer; the recipient must satisfy writer access, canonical resolver exclusions, and task review policy under row locks. - The shared validator export barrel crosses 1,000 lines because it exports the new attention schemas. Schema implementation stays in the dedicated attention validator module. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (OpenAPI contracts and docs/api/attention.md) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
9dfdeaad86 |
Refine agent workspaces and organization charts (#20973)
Refine agent details, identity editing, responsive controls, and organization charts with three card tiers, clearer save feedback, and stable filtering. Guard instruction-file renames and root configuration updates in one transaction through the agent lifecycle integration, preserving conflict checks, rollback, and configuration history. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
45170b0cac | Reuse Runner plugin capability checks (#20987) | ||
|
|
5cd5adfccc |
fix(native): link a person-owned agent blocker to the agent's open question whatever owner kind the model reports (#20969)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - #20931 lets a person's answer continue a task that its agent blocked
while it waited for that answer
> - That link is recorded only when the blocker's unblock descriptor is
bound to the board
> - Codex reports such a wait as `paperclip_block` with `owner: {"kind":
"user", "name": "Board"}` and scope `task_wide`. The evidence classifier
treats only `kind: "board"` (not a PRP owner kind) as board-owned, so
the arbiter binds this person-owned blocker to the reporting agent, and
no link is recorded
> - So every Codex W2Q task still loses the answer
(`issue_not_in_progress`) and needs Retry
> - This pull request classifies a `user` owner as a person
(board-bound) and records the link from the reported owner kind, not
from the spelling of the owner
> - The benefit is that the person's answer reaches the agent whatever
owner shape the model uses
## Linked Issues or Issue Description
Refs #20931
**What happened?**
Restart-chaos wave w-20261111T230717-s5201 on master
|
||
|
|
87fe4d9793 |
fix(workspaces): retry a workspace_finalize record write that loses a deadlock (#20971)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a run ends, Paperclip records a `workspace_finalize` workspace operation. Dependent work reads that record to know if the workspace is safe to use. > - The record insert has foreign keys to `execution_workspaces`, `companies`, `heartbeat_runs` and `issues`. Postgres takes a key-share lock on each parent row, in trigger order: workspace, company, run, issue. > - The native finalizer and other writers lock the issue row and then the run row (the order that #20819 documents). When such a transaction holds the issue and then asks for the run, while the finalize insert holds the run and waits for the issue, Postgres detects a deadlock and aborts one side. > - In restart-chaos wave `w-20261111T205259-s6301` (task t01, run `e7129ec7`) the insert lost. The log says "failed to record workspace_finalize=failed operation; dependents may remain gated". The failed-finalize record was lost. > - The insert's lock order comes from the foreign keys, so the code cannot reorder it without a new explicit lock order that other writers (for example the remote-workspace release path, workspace then run) could invert again. A deadlock victim is safe to run again, because its transaction rolled back completely. > - This pull request retries the finalize record write on lock timeout, deadlock and serialization failure (55P03, 40P01, 40001), with the same bounded policy that #20850 added for the provider-pack admission write. > - The benefit is that a short lock conflict can no longer delete the record that releases dependent work. ## Linked Issues or Issue Description **What happened?** Run `e7129ec7` failed its native session recovery (`runner_harness_state_mismatch: backup_without_reusable_lease`). The failure path called `recordWorkspaceFinalize("failed")`. The insert into `workspace_operations` failed with `PostgresError: deadlock detected` (40P01). The server logged that dependents may remain gated. The run then settled as failed with `native_session_interrupted`; the next `finalizeNativeRun` call logged `native_finalization_missing` because that run had no proposed result. **Expected behavior** A transient deadlock does not lose the finalize record. The write runs again and the record is saved. **Steps to reproduce** 1. Transaction A: lock the issue row `FOR UPDATE`. 2. Start a `workspace_finalize` record for a run on that issue. Its insert takes key-share on the workspace, company and run rows, then waits for the issue. 3. Transaction A: lock the run row `FOR UPDATE`. 4. Postgres reports a deadlock. Before this change the insert is the victim and the record is lost. **Additional context** The new test reproduces this with two real transactions on embedded Postgres. It gives transaction A a longer `deadlock_timeout`, so the insert is always the victim, as on the chaos run. Refs #20850 Refs #20819 ## What Changed - New `server/src/services/transient-lock-contention.ts`: `postgresErrorCode`, `isTransientLockContention` and `retryTransientLockContention` (5 attempts, 100 ms × attempt backoff, only 55P03, 40P01 and 40001). - `workspaceOperationService.createRecorder().recordOperation` wraps its `workspace_finalize` state writes (the running-row insert and the terminal update) in that retry. Each attempt is a new transaction, or a new savepoint when the service runs inside a caller's transaction. - `native-provider-pack-admission.ts` uses the shared helper. Its behavior and its exported `nativeProviderPackAdmissionFailureCode` name do not change. - New test in `server/src/__tests__/workspace-operations-reconciliation.test.ts`: a real two-transaction deadlock between an issue-then-run locker and the finalize insert. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/workspace-operations-reconciliation.test.ts -t "loses a deadlock"` failed with `deadlock detected`, code `40P01`, on the same `insert into "workspace_operations"` query as the chaos log. - Green: the same test passes; I ran it 3 times. The full file and the provider-pack admission tests pass (49 tests). `native-result-commit-work.test.ts` passes. - `npx tsc --noEmit -p server` passes. - I stopped `pnpm verify` after it selected 2533 test files; required CI supplies broad coverage. - Chaos: no new cell. The existing W2 × K1 cell (Codex, Daytona warm) reached this path when the restart fault broke native session recovery. I did not rerun a wave; both wave slots were busy. ## Risks - Low risk. Only the `workspace_finalize` state writes retry, and only on the three contention codes. Other errors throw as before. - A deadlock victim's transaction is fully rolled back, so the retried insert (same operation id) and the retried update are idempotent. - The worst added delay is about 1 s of backoff across 5 attempts, on the failure path only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
827bd21017 |
fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy
## Linked Issues or Issue Description
Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).
**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.
**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.
**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.
**Paperclip version or commit**
master at
|
||
|
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
|
9c56e5465c |
fix(native): keep the agent's own unanswered question open after a turn ends without a report (#20934)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question and wait for the answer. Native task status comes from the server's status arbiter, not from model prose > - When a native turn ends without `paperclip_finish`, the Runner sends a default report: `done`, objective satisfied, with the agent's final message as the only evidence > - A later comment from the person moves an open question into history (#15392), so the question no longer gates completion > - Together these close a task as `done` while the agent is still waiting for its own answer and has produced nothing. No Retry appears and nothing flags it > - This pull request keeps such a task `in_review` on the agent's own unanswered question when the only completion evidence is the agent's final message > - The benefit is that a "waiting for your answer" turn can no longer become a silent false completion ## Linked Issues or Issue Description Refs #15392 **What happened?** Restart-chaos wave on master |