PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-13 15:55:46 +01:00

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	2e6850e745	perf(task-chat): open tasks from a preloaded cache and cold tasks in one request (#20592 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open tasks from task lists, the Inbox and the sidebar many times each hour > - A task's chat shows its first frame only after about seven separate reads finish > - #20355 added one batched `chat-bootstrap` read that returns all of these reads. #20450 used it only for the sidebar's Recents > - The hosted database is CPU-bound, so the UI must open tasks faster and must also send fewer requests > - This pull request uses the bootstrap for task lists, for open intent on any task link, for cold opens and for a hard load of a task URL. It shares one coordination module between all of these paths > - The benefit is that a task opens from cache more often, a cold open sends one chat request instead of about seven, and the first content paint comes sooner ## Linked Issues or Issue Description Refs #20355 (server read that this pull request consumes). Refs #20450 (Recents preload, now on the shared path). Is your feature request related to a problem? A click on a task in a task list waited for about seven separate chat reads. The default task list (Coordinator) had no preload at all. A press or a resting hover on a task link sent the same seven reads, and the page then sent a second set. A hard load of a task URL started its chat reads only after the JavaScript bundle and the company list loaded. Sidebar project task links had no open-intent prefetch. Describe the solution you'd like Use the single `chat-bootstrap` read everywhere a task is about to open. Let the page's gating reads share a bootstrap that is in flight instead of sending their own requests. Keep the core issue read separate so the header does not wait for the larger read. Describe alternatives you've considered Preload 30 rows per list in three requests (the first prototype). Each task costs the server about fifty queries, and a resting hover warms any other row before the click, so this change preloads 10 rows in one request. ## What Changed - `ui/src/lib/issue-chat-bootstrap.ts`: one module decides when to send a bootstrap and lets page reads share it (`readFromIssueChatBootstrap`). Seeding uses the existing `issueChatBootstrapCache` (replace-style reads are refreshed when newer, merge-style reads are filled only when empty, other companies are ignored). It skips tasks that are loading, loaded in the last 30 seconds, or already fresh in cache, under the id or the identifier. It never aborts a request in flight, so a click that unmounts the list still uses the answer. Each request is bound to the account that sent it; a request sent before the first session read is bound to the first account the session reports, and a sign-out revokes it. A page read shares a bootstrap only while its key holds no data (a refresh after a write always reads the server), and waits at most 5 seconds for it. Every ref of a task (id and identifier) is tracked at request start. Entries expire after 30 seconds. - Task lists preload the chat of their top 10 rows in one request, after the list's own reads settle and the browser is idle (`usePreloadTaskChats`, `when-page-settles.ts`): the Coordinator task list (`TaskList` with `preloadCompanyId`), the streamlined and legacy issues lists and the streamlined Inbox. Recents now uses the same path, so a task in both places loads once. A list that stays open preloads each task at most once in five minutes: live events (#20563's lanes) keep a preloaded task's cache current or mark it stale, and a stale key is read from the server, never from an older bootstrap. All seeded keys are per-task keys on the fast live lane, not the ambient lane. - Open intent (press, click, resting hover or focus) on a task with no chat history in cache warms it with one bootstrap request instead of six separate reads (`prefetchIssueDetailForNavigation`). A task seen before still refreshes its cached reads one by one. - A cold open starts one bootstrap. The gating reads (recent comments, activity, runs, input facts, interactions, attachments, live runs, active run) wait for it instead of sending their own requests. The core issue read stays independent. - A hard load of a task URL starts `GET /api/issues/:ref/chat-bootstrap` from the inline preload script, in parallel with the JavaScript bundle. The page adopts that answer. - Sidebar project task links (Coordinator sidebar) get the same hover, press and focus warming as Recents and `IssueLinkQuicklook`. - Tests: unit tests for the coordination module, Recents, `TaskList` preload, sidebar warming and the cold-intent path. A new layout journey checks that a task the Inbox preloaded reveals once at its latest message without reading its chat again. Journeys that hold one separate read now also hold or fail the bootstrap. The Inbox reveal journey held a read that the chat no longer sends on open (it was flaky on master too), so it now holds the recent comment window. - The perf harness gains `PAPERCLIP_ISSUE_PERF_LIST_DWELL_MS` and the README documents it. ## Verification Measurement setup: production UI builds of master (`e01fe7f84a` and the earlier base `8d2bf78323`) and of this branch, served by one local server with `PAPERCLIP_UI_DEV_MIDDLEWARE=false` and embedded PostgreSQL. Only `index.html` changed between variants (both asset sets were present), so the server, database and seeded task were the same. Four rounds of the issue-detail harness, with master and branch alternated in each round: 20 pairs per S1 cell, 40 per S2 cell. The host was shared with other agents (load average 16 to 43), so the spread is large. Content paint (the task is readable), median paired delta, branch minus master: \| Scenario \| Profile \| Pairs \| Content paint \| Branch faster in \| API requests before content (master / branch) \| \|---\|---\|---:\|---:\|---:\|---:\| \| S1 preloaded click (dwell 1500 ms) \| unthrottled \| 20 \| -45 ms \| 15/20 \| 25 / 18 \| \| S1 preloaded click (dwell 1500 ms) \| Fast 4G, 4x CPU \| 20 \| -23 ms \| 13/20 \| 26 / 18 \| \| S1 click, no dwell \| unthrottled \| 20 \| -51 ms \| 15/20 \| 25 / 18 \| \| S1 click, no dwell \| Fast 4G, 4x CPU \| 20 \| -21 ms \| 12/20 \| 26 / 18 \| \| S2 cold deep link \| unthrottled \| 40 \| -61 ms \| 28/40 \| 54 / 40 \| \| S2 cold deep link \| Fast 4G, 4x CPU \| 40 \| -129 ms \| 29/40 \| 47 / 39 \| Medians (p25 to p75), content paint in ms: S1 dwell unthrottled 295 (283-315) → 248 (217-276). S1 dwell Fast 4G 1133 (1114-1226) → 1093 (853-1176). S1 no dwell unthrottled 295 (277-328) → 260 (233-332). S1 no dwell Fast 4G 984 (842-1196) → 923 (816-1110). S2 unthrottled 1002 (677-1235) → 896 (619-1280). S2 Fast 4G 4643 (4155-5061) → 4516 (4298-4894). Header paint is neutral to slightly later in two cells: S1 dwell unthrottled +39 ms (3/20 faster) and S2 Fast 4G +56 ms (17/40). In those cells the header now paints in the same frame as the content. In the first two rounds, the median gap from header to content fell from 105-157 ms to 29-74 ms unthrottled and from 557 ms to 298 ms for S2 on Fast 4G. Other header cells are -27 to +1 ms. The local server answers each read in a few milliseconds, so the network wait that the preload removes is small here. The request count is the stable result: a cold open sends one chat request instead of about seven. List page: the preload request starts after the first task row is in the DOM (64 ms after it unthrottled, 200-280 ms after it on Fast 4G, page clock). Time to the first visible row, 30-task list: unthrottled master 1863 ms vs branch 1563 ms (6 pairs, branch faster in 5). Fast 4G master 3626 ms vs branch 3727 ms (14 pairs, branch faster in 6, mixed signs). The preload starts after the row paints, so it cannot delay it; the Fast 4G difference is noise. One 10-task bootstrap took 78-162 ms on the server. Checks run: - `pnpm exec vitest run` for the bootstrap, cache, prefetch, Recents, sidebar, quicklook, `TaskList`, issues list, Inbox, issue detail and Coordinator tests: 18 files, 646 tests pass. - `pnpm exec tsc -p ui/tsconfig.json --noEmit`: clean, except missing generated Storybook catalog files that this change does not touch. - `pnpm check:token-gates`: clean. - `tests/e2e/task-loading-frame.spec.ts`, `disconnect-resilience.spec.ts` and `connection-reviews.spec.ts` (production build): 21 passed, 1 skipped. - `tests/perf/issue-detail/layout-stability.spec.ts` and `plan-startup.spec.ts` against the production build: all pass. The Inbox reveal journey and the new preloaded journey passed 4 of 4 repeats each. - `pnpm verify` asks for broader CI coverage because an e2e spec changed. ## Risks - More server work for list views: one bootstrap of up to 10 tasks per list visit (about fifty queries per task), skipped for tasks already loaded in the last 30 seconds. It runs only after the page's own reads settle and the browser is idle, and only while the tab is visible. - A resting hover on a never-opened task now sends one bootstrap instead of about six reads. The server work per task is about the same; the request count is lower. - If a bootstrap fails, each page read falls back to its own request. The fallback is slower than before for that one open (the reads start when the page mounts, not at the press). - The header can paint a few tens of milliseconds later in the same frame as the content when everything is cached. - An answer that arrives after a sign-out or account change is dropped. An answer for another company is ignored. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Playwright runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 14:26:27 -06:00
Devin FoleyandPaperclip	8148fc0b1e	fix(release): fetch draft-notes PR summaries in batches (#20602 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The release workflow drafts the next stable's release notes when a beta publishes (`draft_stable_notes`) > - The draft nests each referenced PR's summary under its commit subject, and it fetches each summary with its own `gh pr view` call > - A beta range now references more than 1,500 PRs, so these serial calls do not finish inside the job's 10-minute timeout > - This pull request fetches the summaries in batches of 50 per GraphQL query > - The benefit is that the job finishes in under a minute and pushes the notes branch again ## Linked Issues or Issue Description What happened? The `draft_stable_notes` job of [run 42763789753](https://github.com/paperclipai/paperclip/actions/runs/42763789753) (beta `2026.1110.0-beta.0`) reached its 10-minute timeout. GitHub cancelled it during "Draft stable notes from the published beta", while a `gh` process was still running. The beta itself published and its smoke tests passed. The `release-notes/v2026.1110.0-beta.0` branch was not pushed. Expected behavior The job writes the notes draft and pushes the branch for every beta, whatever the size of the commit range. Steps to reproduce 1. Publish a beta whose range from the last stable has many PR-referencing commits. This range has 1,586 commits, and 1,583 different PRs. 2. `draft-stable-notes.sh` calls `gh pr view <n>` once for each commit, one after another. 3. The job reaches `timeout-minutes: 10` before the loop ends. Paperclip version or commit `09e24dd12c36251dd12af2d73dbc715ce990f922` (beta `2026.1110.0-beta.0`). This was the first beta to run the job after #12124 added the PR summaries; earlier betas skipped it. ## What Changed - `scripts/draft-stable-notes.sh`: - Collects the unique PR numbers first. Fetches their bodies 50 per `gh api graphql` query (aliased `pullRequest(number:)` fields), into a temporary cache that `enrich_pr` reads. - Parses the raw response with `jq`, not with `gh --jq`. When any number in a batch is not a PR (for example, an issue reference), gh exits non-zero and does not apply its `--jq` filter. The other PRs in the batch still have usable data. - The output format does not change. A missing `gh` or `jq`, a failed batch, a cache directory that `mktemp` cannot create, or `DRAFT_NOTES_SKIP_PR_ENRICHMENT=1` still gives bare subjects. - No code comments are added (per `CONTRIBUTING.md`); this description holds the explanation. - `scripts/draft-stable-notes.test.mjs`: - The enrichment tests now use a fake `gh` that answers GraphQL like GitHub. - New tests: batching (120 PRs make 3 queries, and a repeated PR is fetched once); a batch with a non-PR number; a failed batch; a failed `mktemp`; prose-only and empty bodies; the opt-out. - Range tests now run with enrichment disabled, so an authenticated `gh` on the test machine cannot reach GitHub. - `package.json`: adds the test file to `test:release-registry`. These tests did not run in CI before. ## Verification - `node --test scripts/draft-stable-notes.test.mjs`: 17 pass. They also pass with macOS bash 3.2 first on `PATH`. - `node scripts/ci-profile/node-test.mjs scripts/draft-stable-notes.test.mjs scripts/release-lib.test.mjs`: 31 pass. - Red check: the new tests against the old script fail 5 cases (all batching and enrichment cases). The `mktemp` test fails against an unguarded `mktemp` call. - Live run on the real range: `./scripts/draft-stable-notes.sh 2026.1110.0-beta.0` finished in 47 seconds. It wrote 4,596 excerpt lines for 1,583 PRs. - Parity: for 40 randomly sampled PRs from that range, the new excerpts are byte-identical to the old per-PR `gh pr view` output. - `shellcheck`: no new findings. The two remaining notices were there before this change. ## Risks - Low. The change touches only the post-publish draft job, and every failure still gives the bare-subject draft. - `jq` is now needed for the PR summaries. GitHub's Ubuntu runners include it. Without `jq`, the draft has no summaries, the same as without `gh`. - I did not change `release.yml`. The job already checks out `master`, so after this merges, re-running the failed job of run 42763789753 should push the missing notes branch with the fixed script. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use (shell, file editing, test runs, read-only GitHub API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 11:48:55 -08:00
DottaandClaude	90dd864bff	test(restart-chaos): wait for any visible Retry, not the first match (#20616 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness checks that a task never needs a human Retry after a control-plane restart, and that every Retry the board shows works > - When a task rests with a Retry offer, the harness opens the task page and clicks the Retry button like a user > - The harness waited for the first `Retry` button in document order to become visible > - A page can hold an earlier `Retry` button that is in the accessibility tree but has no box. The harness then waited on that button for the full timeout and reported `ui_retry_button_missing`, although the user saw a Retry in the status banner > - This pull request makes the harness wait for any visible Retry button, banner first > - The benefit is that a chaos grade reports a missing Retry only when the user cannot see one ## Linked Issues or Issue Description What happened? A Daytona chaos wave graded one task `ui_retry_button_missing`. The captured status banner text was "Couldn't finish this attempt. Your message is saved. Retry", so the page showed a Retry button. The harness waited on `getByRole("button", { name: "Retry", exact: true }).first()`. That locator selects the first match in document order. When that match is not visible, the wait times out even when another Retry is visible. The click loop after the wait already used only visible buttons, but the wait itself did not. Expected behavior The harness reports `ui_retry_button_missing` only when no visible Retry button exists before the timeout. When a visible Retry exists, the harness clicks it, banner first. Steps to reproduce 1. Serve a page with a zero-size `Retry` button before a visible status banner that holds a second `Retry` button. 2. Call `BoardUi.clickRetry` for that page. 3. Before this change, the result is `ui_retry_button_missing`. After it, the harness clicks the banner Retry and sees the retry request. Separate finding, not changed here: in that wave the Retry was also a dead end on the server side. The run had failed with `native_session_cleanup_quarantined`. The server build that ran the wave still projected `canRetry: true` for that run, and the retry route refuses that code with 409. Current master already removes that Retry offer (#20572). ## What Changed - `tests/restart-chaos/board-ui.ts`: wait on the `Retry` role locator filtered to visible elements (`filter({ visible: true })`), instead of the first match in document order. After the wait, re-query the visible buttons and click them banner first, as before. - `tests/restart-chaos/board-ui.test.ts`: add a page with an earlier zero-size `Retry` and a visible banner `Retry`. The test fails before the change (`ui_retry_button_missing`) and passes after it. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/board-ui.test.ts tests/restart-chaos/triggers.test.ts`: 43 passed. With the old `board-ui.ts`, the new test fails with `ui_retry_button_missing`. - `pnpm exec tsc --noEmit -p tests/restart-chaos`: passes. ## Risks - Low risk. The change is test-harness code only. No product code changes. - The timeout and the "missing" result do not change. The locator `visible` filter needs Playwright 1.51 or later; the repo uses 1.62. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 13:40:31 -06:00
DottaandClaude Opus 5.5	5f34f777ec	fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog checks that a chat agent does the work a person asks for > - In the latest full-cohort run on master, 16 cases failed although the saved work was correct > - Their graders matched words, one literal value, or one API route in free-form model output > - A grader that fails correct work hides real failures in the noise, and it can also pass wrong work that uses the expected word > - This pull request grades those cases by saved state, audit records, setup literals and rendered page structure, and keeps only narrow checks for false claims > - The benefit is that these cases fail only when the product or agent does the wrong thing ## Linked Issues or Issue Description Refs #18971 (related: company-secret question handoffs) What happened The full-cohort chat-eval run on master failed 16 cases where the retained evidence shows correct behaviour. Examples: `companies.rename-company` accepted only `company.updated`, but the supported branding route logs `company.branding_updated`. `recovery.post-stop-message` required a signup button label that matched a regex, and rejected a labelled email form with the button "Start writing free". `secrets.secret-proposal` flagged "the value is delivered when Morgan next runs" as a false delivery claim. Other graders required words such as "uncommitted", "telemetry", "administrator" or the app name that the question did not ask for. Expected behavior A grader accepts every supported route and wording that achieves the outcome. It fails when saved state is wrong, when a requested fact is wrong, or when the reply makes a claim that saved state contradicts. Steps to reproduce 1. Load the retained observation for one of the cases below from the full-cohort evidence. 2. Grade it with the old oracle. The keyword or route check fails although the saved state is correct. 3. Grade it with the new oracle. It passes. A broken variant (wrong saved state or a contradicting claim) still fails. ## What Changed - `activity.audit-change`: attribution is graded by structure. The reply names the recorded editor, names no other company agent (new setup literal `otherActors`), and states only real transitions. Sentence grammar is no longer matched. - `app-permissions.ask-before-action`: each continuation reply must exist and must link no app record other than the reviewed connection. It no longer must name the app. - `budgets-costs.spend-breakdown`: the reply must point at the record with unresolved usage (task link, identifier or run id) and must not give it a zero or free amount. The words "subscription" and "incomplete" are no longer required. - `cases.browse-cases`: an excluded case may be mentioned in prose. It must not be linked or listed as a result row. - `cases.handoff-same-task`: handoff instructions carry findings forward when they point at the saved `findings` document, reproduce it, or restate it. The premise (a primary Morgan run performs the handoff) is unchanged. - `cases.update-case-facts`: a saved browser value may add clauses that only rule out other platforms ("desktop Safari unaffected"). - `companies.agent-observability`: the reply no longer must name Telemetry and the run log. It must not claim that they are exported. - `companies.rename-company`: both supported name routes (`company.updated`, `company.branding_updated`) are accepted. - `connection-setup.choose-provider`: the answer must exist and must not claim a connected account. The unavailable method no longer must be named; the setup path stays graded by clicking the card or link. - `goals.align-existing-work`: a question card from the answering run counts as the phase-two response. Refusal words and "unchanged" words are no longer required; false change claims still fail. - `recovery.post-stop-message` (and `steering.interrupt-with-direction`, which shares the helper): a visible form with a labelled field and a submit control is a signup action, whatever the button says. - `routines.pause-owned`: the "paused"/"running" words are no longer required. The saved checkpoint proves both; a cancellation claim still fails. - `secrets.secret-proposal` (and `secrets.configure-company-secret`, which shares the helper): a sentence that says when delivery will happen is not a delivery claim. - `skills.explicit-restriction`: the reply no longer must say "administrator". Success claims and invented tool limitations still fail. - `workspaces.inspect-git`: the reply no longer must say "uncommitted". Unchanged HEAD and status prove it; commit or push claims still fail. - Every affected `oracleVersion` is bumped in code and in `chat-catalog-descriptors.json`. No scenario YAML changed. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. - `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact canary with Chromium: pass, including new form cases. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged pins. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Offline calibration on retained evidence, for each case: the retained correct evidence passes, earlier passing evidence passes where it exists, and a broken variant fails. `spend-breakdown` retained evidence still fails, because its reply shows unresolved usage as `$0.00`. `handoff-same-task` retained evidence still fails on the kept primary-run premise. - Live paid runs: see the comment below. ## Risks - Low risk. The change is limited to eval graders and their unit tests. Product code does not change. - Each grader is more accepting of wording. Saved-state checks and the narrow false-claim checks remain, so a wrong outcome still fails. - `secrets.configure-company-secret` still fails its retained evidence. Its agent asked a two-question card, and the act supports only one-question cards. This pull request changes it only by the shared delivery-claim helper. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 13:37:16 -06:00
DottaandPaperclip	89ec4e7614	feat(links): add cached GitHub pull request previews (#20593 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task conversations contain links to pull requests and other work. > - A URL alone cannot tell the operator whether a PR is open, merged, or blocked by conflicts. > - Fetching every link at render time wastes API calls and can expose private data through a shared cache. > - This pull request adds a demand-driven links service and a GitHub PR adapter. > - Each viewer and company has a separate cache. Repeated links share one request and result. > - Operators get context when they hover or focus a link. Normal links stay useful when a provider is unavailable. ## Linked Issues or Issue Description Subsystem affected Task Markdown links, shared link metadata, and the server GitHub API boundary. Problem or motivation PR links have compact labels and neutral icons. They do not provide a rich hover card or a common status source across repeated mentions. A background resolver cannot borrow a person's GitHub connection because it has no viewer identity. Proposed solution Recognize supported links locally. Request metadata after a delayed hover or keyboard focus. Show a card at the first line of the link, aligned with its icon. Share the result across all mentions in the current viewer and company. Keep the exact original navigation URL. The first adapter supports GitHub pull requests. Open is green, merged is purple, and merge conflicts are orange. Draft, closed, unknown, loading and unavailable states remain quiet. Alternatives considered Eager page scans would spend API quota on links the operator does not inspect. Company-wide caching of personal GitHub results would cross the credential boundary. This change uses neither approach. Roadmap alignment ROADMAP.md was checked. This extends existing task links and the GitHub resolver. It adds no deployment adapter or new background worker. Searches of open and closed PRs for link previews and PR hover cards found no matching implementation. ## What Changed - Add host-owned link recognition and a normalized preview contract. The service selects an adapter by provider. - Add a board-only company read endpoint. It checks company access before cache reads and sends a private, no-store response. - Use the current viewer's eligible GitHub connection. A configured connection that cannot supply credentials fails closed. With no managed connection, reuse existing company credentials or public access. Instance credentials retain the public-repository check. - Reuse the GitHub external-object resolver for API transport and response parsing. Add title and mergeability to its normalized snapshot without changing its existing state contract. - Coalesce requests by company, viewer and canonical PR URL. Bound cached entries and concurrent work. Cache success for one minute and failures for a short period or the provider retry window. - Add an abortable request deadline. Timed-out work stays in the concurrency budget until it settles. No periodic worker or idle polling is added. - Add a shared browser query and anchored preview card. Repeated Markdown links observe the same result. There are no automatic retries, window-focus refreshes, toasts or connection prompts. - Add ten production-component Storybook examples and real pointer/keyboard browser checks. Document the API and verification route. ## Verification - Linux source checkout: 233 focused tests passed in nine suites. They cover recognition, fixed API destinations, provider states, access failures, credential selection, viewer/company isolation, request coalescing, expiry, cache bounds, concurrency bounds and mounted OpenAPI coverage. - A live public GitHub smoke returned this PR's correct title and open state. Two equivalent URLs shared one real network request. Personal connection access was not part of that smoke. - A controlled HTTP server stalls a real response body. The test proves cancellation while another lookup completes. - UI typecheck and the server layered typecheck passed. The UI production build passed. - Token gates and query policy gates passed. All 30 map/catalog support checks passed. The three new server test files also passed a focused TypeScript check. - All 27 focused browser checks passed on the reviewed source tree. They cover light/dark themes, narrow layouts, open/merged/conflict/draft/closed/loading/unavailable/failed states, fixed positioning during pointer movement, exact navigation and keyboard access. A real browser HTTP interception counts one deferred lookup for two equivalent PR links. - The first browser attempt lacked Chromium. A later combined build/test attempt exhausted the sandbox's 8 GiB memory limit and killed Storybook. Those attempts supply no pass evidence. Chromium is installed, and the final browser checks run with the build stopped. One initial Storybook dependency reload interrupted a streaming check. All eight existing inline-link cases passed on the warmed server, followed by a complete 27-case pass. - Synthetic Storybook pointer events did not exercise the native hover boundary. The assertions now run through real Playwright pointer and keyboard actions. The fixture stories retain the production components. - Review fixes preserve the Draft label when GitHub reports both draft and conflicts. They also preserve neutral link colors in dark mode. The 25 affected service checks pass, including the added combined-state regression. Browser checks compare unknown, draft and closed links with the muted theme color. - The first CI policy run rejected a literal retry opt-out. The code now uses the existing explicit zero-retry helper. A new test-spy type error was also corrected. These failures are fixed; their earlier runs remain failed. - `pnpm verify --explain` returned exit 2 because the browser test path selects broad CI. No full local suite is claimed. The final CI run supplies that coverage. - The previous head `437913c797963aa113bb82e2ae8b836969317ef9` passed CI and received a 5/5 Greptile score. All four review threads are resolved. Master then advanced. The current head `ae2a1c2adbd61a621cca9bdd160561952613df6e` resolves the source-manifest conflict against master `9b21dcbdda01296349f3cfe0a3a6891e79df6130`. Fresh CI and review are running. - The previous CI run found a lost class on decorated PR links and a stale SDK source manifest. Both are fixed. The existing CommentThread integration test and 26 focused link service tests pass. All 45 SDK source/privacy tests pass. All eight source-manifest helper tests pass. The manifest helper changed only source identities and added the two new shared files. Runtime, compiler, parser and library pins stay unchanged. - A later review found that expired timeouts could reuse a settled promise. The service now removes settled requests from its shared-request map and counts unfinished provider work separately. The new regression failed before the fix and passes after it. The concurrency-bound and real stalled-HTTP tests also pass. - UI typecheck and token gates pass after these fixes. The browser rerun needed Chromium in the new session. That first attempt did not exercise the UI. The complete 27-case browser rerun passed before the master merge with retries disabled. All 27 browser cases also passed on the merge revision with retries disabled. - After the master merge, all 84 focused link, OpenAPI and integration tests pass. All 55 SDK source/privacy/manifest tests pass. UI typecheck and token gates pass again. The source manifest retains master's reviewed runtime pins and includes both sets of source changes. GitHub confirms the PR can merge cleanly. The preflight helper rebuilt the shared package and plugin SDK; Cargo is unavailable locally. Final CI supplies Rust coverage. - The latest master update changed the reviewed report-lock policy but left the SDK lock hash at its previous value. The SDK source/evidence checks reproduced the mismatch. This change sets only the SDK lock hash to the authenticated output of the existing master policy and patch. Compiler, parser and Zod pins stay unchanged. All 56 SDK source/evidence/manifest tests now pass. All 84 focused link/API/integration tests, UI typecheck and token gates also pass on the updated source. ## Risks - This cache is process-local. Separate server replicas can make separate GitHub requests. - A cached result can remain visible for its short lifetime after GitHub changes. A later hover refreshes expired data. - GitHub can report unknown mergeability. The UI shows conflicts only when GitHub explicitly reports them. - Personal preview data stays in viewer/company caches. It is not written to company external-object records or persisted browser query storage. - Stories simulate the API. They do not prove a live GitHub login or private-repository access. - No schema migration is required. ## Model Used OpenAI Codex, based on GPT-6. The runtime does not expose the exact serving model ID or context-window limit. Used reasoning, repository inspection, source editing, shell execution and browser tools. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 19:32:51 +00:00
DottaandClaude Opus 5.5	42d4499489	fix(daytona): force a stuck stop before starting a retained sandbox (#20609 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Daytona, a follow-up task resumes the agent's retained sandbox before the agent starts > - The host stops an idle sandbox gracefully. A process that ignores SIGTERM can leave Daytona in `stopping` for many minutes > - Resume waited for that state change until its 300 s start deadline. It never escalated, so each attempt failed after five minutes, and the bounded retries did the same > - On staging, a user comment waited about 21 minutes before the agent started. Two resume attempts each used 299.7 s in the plugin dispatch > - This pull request forces a stuck stop once, then starts the sandbox within the same deadline. It also traces the wait with the observed state > - The benefit is that a stuck stop costs about 30 s instead of failing the run, and the next case is diagnosable from the trace ## Linked Issues or Issue Description What happened? A user commented on a task that used a retained Daytona sandbox. The first run and its first retry each failed after about 5 min 20 s with `reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no `lease.start`. So the whole budget went to `waitOutSandboxTransition`, which polled a sandbox that stayed in a transition state. The later attempt, after a control-plane restart, found the sandbox `stopped`. Its `lease.start` took 1.3 s. A transition that ends in `stopped` is a stop. Expected behavior A sandbox that stays `stopping` for longer than the host's own graceful-stop budget is forced to stop, then starts. The run does not fail on the start deadline. Steps to reproduce 1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona reports `stopping`). 2. Resume or realize the lease. 3. Before this change, the call polls until the 300 s deadline and fails with "is still stopping at its start deadline". ## What Changed - `packages/plugins/sandbox-providers/daytona/src/plugin.ts`: `waitOutSandboxTransition` tracks how long the sandbox stays `stopping`. After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls `sandbox.stop(seconds, true)` once, then keeps polling. A refused force stop keeps the old wait within the same deadline. - The wait is a `lease.wait_transition` span. The force stop is a `lease.force_stop` child span. - `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label also records `starting`, `stopping`, `restoring`, `archiving` and `resizing`. Before, these were all `other`. - `server/src/services/plugin-host-services.ts`: the host admits the two new span names and the new closed state values. - `doc/observability.md`: documents the new spans and states. ## Verification - Red/green: with the plugin change stashed, the three new stuck-`stopping` tests (realize, refused force stop, resume) fail by hanging to the deadline. With the change they pass. The "settles on its own" test confirms that no force stop happens for a normal stop. - `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14 files, 618 tests passed (8 skipped). `tsc --noEmit` passed. - `server/src/__tests__/plugin-host-services-span.test.ts`: red before the allowlist change, 25/25 after. - `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies the broad coverage. - Not verified on a live Daytona sandbox. Hosted acceptance needs the change deployed to staging. ## Risks - The force stop sends SIGKILL to processes in a sandbox that is already stopping and is about to start. The disk persists across a stop, so the retained workspace is kept. A graceful stop that is still running past 30 s is killed. The host's own graceful stop budget is 30 s, so the host had already given up on it. - Daytona can refuse a stop while a state change is in progress. Then the code waits as before, so the result is no worse than today. - The root cause is inferred from the trace (a transition that ends in `stopped`). The new state label confirms or refutes it the next time it happens. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 19:20:30 +00:00
DottaandClaude Opus 5.5	9b21dcbdda	refactor: remove SPEC docs and repoint spec citations (#20565 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents read the docs under `doc/` before they change code > - `doc/SPEC.md` and `doc/SPEC-implementation.md` were out of date. `doc/SPEC-implementation.md` became an append-only log: in 60 days it got about 1,645 new lines and lost 181, across 140 commits > - #20557 changed `AGENTS.md` so the code and its tests are the source of truth, and the specs are only background > - This pull request removes the two spec docs. It moves only the sections that live code comments cite into focused docs, and it repoints all other references > - The benefit is one less stale document that agents read and append to ## Linked Issues or Issue Description Refs #20557 (merged). This PR is the follow-up that removes the docs. It is based on `master`, which already contains #20557. Issue type Outdated content. The docs are out of date and conflict with the code. Where is the issue? `doc/SPEC.md`, `doc/SPEC-implementation.md`, and the code comments and docs that cite them. What's wrong? `doc/SPEC.md` and `doc/SPEC-implementation.md` are out of date, but agents still read them and append a paragraph with each fix. Suggested fix Delete both files. Move the sections that code cites into focused docs. Repoint the references. ## What Changed - Deleted `doc/SPEC.md` and `doc/SPEC-implementation.md`. - Moved these spec sections, because live code comments cite them: - §9.3.1 "Shared default-open issue writes" → new section in `doc/agent-permission-defaults.md`. - §9.8.1 "Issue-thread interaction resolver contract" → `doc/execution-semantics.md`, "Issue-thread interaction resolution". That section already had most of the contract. I added only the missing parts: the `supersedeOnUserComment` default, the canonical policy list, the addressee and cap rules, the rule that surfaces show the server's effective audience, and the cross-issue budget charge. The short policy summary at the top of the section now points to the list, so the definitions occur one time only. - §9.9 "Task Watchdog Authority Contract" → new "Authority contract" section in `doc/TASK-WATCHDOG.md`. It holds the allowed and disallowed mutation lists and the deferred capabilities. The atomic recovery batch, restoration verification, and interaction rules were already in `doc/execution-semantics.md` §11, so the new section links there and does not repeat them. I did not move the "Downstream acceptance criteria" checklist or the sentence about the CTO and Security sign-off. These were implementation-time notes; the tests now hold that coverage. - §9.10 "Company Skill Policy Contract" → new `doc/company-skill-policy.md`. No focused doc had this contract. I did not move the "Required regression tests" phase checklist. - Did not move any other part of the specs. - Repointed code comments that cited spec sections: - `packages/shared/src/issue-write-denial.ts` (§9.3) → `doc/agent-permission-defaults.md` - `server/src/routes/issues.ts` (§9.9) → `doc/TASK-WATCHDOG.md`, "Authority contract" - `server/src/routes/issues.ts` (§9.3, §9.8.1 cross-issue budget) → `doc/agent-permission-defaults.md` - `ui/src/lib/interaction-audience.ts` and `ui/src/components/InteractionGovernancePanel.tsx` (§9.8.1) → `doc/execution-semantics.md` - `ui/src/lib/skill-policy-denial.ts`, `ui/src/pages/CompanySkills.tsx`, `ui/src/pages/CompanySkills.production.tsx`, `ui/src/pages/SkillStudio.tsx` (§9.10) → `doc/company-skill-policy.md` - `server/src/services/recovery/ownership-invariant-report.ts` (monitor deadline wait path) → `doc/execution-semantics.md` §8. That section already lists the one-shot monitor as a valid wait path. - Resynced the git blob pin for `packages/shared/src/issue-write-denial.ts` in `tests/runner-e2e/source-sdk-public-inputs.json`. The comment edit changed the file's hash, and the Runner support suite checks that pin. I used `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`; only that one pin changed. - Updated `AGENTS.md` §1 so it does not name the deleted files. - Removed the links to the deleted files from `doc/plugins/PLUGIN_SPEC.md`. - Updated the `doc-maintenance` agent skill (`.agents/skills/doc-maintenance/`) so it no longer audits or stages `doc/SPEC.md`. - Removed or replaced references in `maps/README.md`, `maps/feature-map/README.md`, `maps/storybook-map/README.md`, `doc/PRODUCT.md`, `doc/CLIPHUB.md`, `doc/DEPLOYMENT-MODES.md`, `doc/DEVELOPING.md`, `doc/evals.md`, `doc/spec/agent-runs.md`, `doc/AGENTCOMPANIES_SPEC_INVENTORY.md`, `doc/connections/CONNECTOR-PLAYBOOK.md`, `doc/plugins/ideas-from-opencode.md`, `doc/execution-semantics.md`, `doc/TASK-WATCHDOG.md`, and `.devin/wiki.json`. - In `doc/execution-semantics.md`, the "admission is independent of cleanup and accounting" reference now states the rule inline: only a hard stop, an explicit pause, or an inactive company refuses admission. - Kept these references as they are: dated plans in `doc/plans/` and `docs/plans/`, the dated log `doc/logs/2026-05-24-cli-api-parity-e2e-log.md`, `releases/`, the transcript fixture `ui/src/fixtures/runTranscriptFixtures.ts` (it uses the filename only as sample text), and `docs/specs/external-task-protocol.md` (it refers to the Symphony project's own `SPEC.md`). ## Verification - `git grep -n "SPEC-implementation\\|SPEC\.md"` (without `releases/`, and ignoring `PLUGIN_SPEC.md` citations) finds only the kept references listed above. - `grep -rn "§9\.$3\\|8\\|9\\|10$" ui/src server/src packages/shared/src` finds no results. - `node scripts/check-maps.mjs`: "Maps aligned: 35 areas, 192 capability IDs". - `pnpm check:token-gates`: "All gates clean". - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts`: 45 passed. These failed in CI before the pin resync. - `pnpm verify`: exit 2 ("Broader verification required") because `AGENTS.md` is a global CI input. Required CI supplies the broad coverage for this revision. - All code changes are comments only. No runtime behavior changes. The title uses `refactor:` because comments in source files changed. ## Risks - Low risk. The change removes docs and edits comments only. - A reader can lose context from spec sections that I did not move. The text stays in git history. The code and its tests are the source of truth. - External links to the two deleted files will return 404. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks above; CI supplies broad coverage) - [x] I have added or updated tests where applicable (not applicable: docs and comments only) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 13:07:59 -06:00
DottaandClaude Opus 5.5	bce79f186f	fix(evals): count a pending question card as the turn's response where the case allows it (#20597 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat agent answers the person and saves the right records > - An agent can end its turn on a question card; the product then records `interaction_owns_conversation_response` and posts no reply comment > - The privileged settle step and some graders still waited for a reply comment, so valid card-ended turns timed out or failed > - This pull request adds one shared rule for when a pending card counts as the turn's response, and uses it where the case allows a card > - The benefit is that card-ended turns are graded on what the person saw, while cases that need a written reply stay strict ## Linked Issues or Issue Description Refs #20588 (request_human_input version fix found by the same runs). Refs #20590 (chat review cards link their document). What happened? In the full chat eval run on master `f4337bb5fb`, agents ended turns on pending question or review cards. `developer-labs.component-reference` timed out in the privileged settle step, which required a reply comment. `goals.align-existing-work` refused the foreign goal in a question card, but the grader read only reply comments. `questions-and-approvals.reject-and-revise` returned its draft in a review card bound to the draft revision, but the grader and the result check required a reply link. Expected behavior A pending card counts as the turn's response only when the product decided that the card owns the run's response, the lead's own run created it, a person can answer it, and it is not a governed approval. Each case still grades the content it requires. Steps to reproduce Regrade the retained evidence of the cases above with the old graders: they fail on the missing reply comment although the card is on the chat. ## What Changed - New `chat-catalog-domains/turn-response-card.ts`: `turnResponseCards()` returns pending cards from runs whose `presentationDecision.reasonCodes` contains `interaction_owns_conversation_response`. A card must be created by the lead, be human-answerable (a text question, two or more choices, or an ordinary confirmation) and not carry `toolAction`, `secretProposal` or `connectionAuthorization`. - `settlePrivilegedChat` treats such a card as a settled turn. The privileged graders still require their written guidance, so `developer-labs.component-reference` now fails at grading instead of timing out. - `goals.align-existing-work` reads the card from the run woken by the foreign-goal request as that phase's answer. A leaked foreign text, or a card that does not refuse, still fails. Oracle `paperclip.chat-catalog.internal.goals-align.v2`. - `questions-and-approvals.reject-and-revise` accepts a review card bound to the draft's exact saved revision. The result check opens the draft from the card's document link (needs #20590). Oracle `paperclip.chat-catalog.plan-review.v6` for both plan-review cases. - Descriptors updated for oracle versions and `sourceFiles`. ## Verification - `turn-response-card.test.ts`: the retained cards pass; other run, other agent, answered, other company, user-created, non-human resolver, single non-text choice, governed confirmation, and a run that replied all fail. - Offline regrade of retained evidence (master `f4337bb5fb` full-cohort run): `goals.align-existing-work` cross-company check passes with the card and fails without it; `reject-and-revise` `real-local-review-draft` passes with the bound card and fails with no card or a card bound to another revision. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains ...clock tests --testTimeout=180000`: 269 files passed (one file needed the plugin SDK build locally, then passed). - Consistency and contract tests pass. `npx tsc --noEmit -p tests/runner-e2e` is clean. SDK public input pins are unchanged. - Paid run [42855262851](https://github.com/paperclipai/paperclip/actions/runs/42855262851) on a temporary branch with this PR, #20588 and #20590 (OpenCode, local): `reject-and-revise` and `revoke-personal-grant` passed; `goals.align-existing-work` failed only `phase-bound-strategic-context-explanation` (keyword check, unchanged here); `developer-labs.component-reference` ended on a question card again and now failed at grading (`existing-question-reference`, `reference-only`) instead of timing out in settle. This run's agents replied in text in the two passing cases, so the card branch of `reject-and-revise` is covered by the offline regrade only. ## Risks - `settlePrivilegedChat` now stops waiting on a card-ended turn. Privileged cases whose graders need a reply fail sooner and with the real failed check; none pass more easily. - Changed oracle versions start new history for `goals.align-existing-work` and both plan-review cases. - The remaining keyword check `phase-bound-strategic-context-explanation` in `goals.align-existing-work` is unchanged and still fails the retained reply; it needs its own fix. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 12:33:52 -06:00
github-actions[bot]andlockfile-bot	9816d2d25c	chore(lockfile): refresh pnpm lockfiles (#20615 ) ## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description What happened? Dependency manifests on master changed after the last committed lock refresh. Expected behavior Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. Steps to reproduce Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>	2026-11-10 18:20:31 +00:00
DottaandPaperclip	12f70d0d4c	Keep Slack form custody and receipt transfer durable (#19825 ) ## Thinking Path > - Paperclip manages work for AI-agent companies. > - Chat connections deliver provider messages and user answers. > - Slack requires a response within its original three-second deadline. > - Forms and their delivery receipts must survive retries and tenant restarts. > - This change uses the existing connector queue and durable form owners. > - Current authorization, exact receipt checks and normal ACK rules stay in force. > - The tenant can settle useful work while keeping the connection enabled. ## Linked Issues or Issue Description Refs #19019 and #19348. Companion Cloud PR: paperclipai/paperclip-cloud#1250. ## What Changed - Preserve form grants, sessions, encrypted evidence, terminal receipts and company-authorized Retry. - Check initial form enrollment before the provider send. Keep a typed preparation wait only while an exact current enrollment obligation exists. - Refund the attempt only when the claimed attempt and credential lease still match and provider invocation has not started. Possible or accepted sends retain uncertain-delivery custody. An ordinary error with the same text still uses the normal five-attempt limit. - Notify the existing publication worker when normal enrollment commits. Reconstruct pending work from durable state after a lost reply or owner restart. - Page settled history before proving idle safety. Keep unfinished, malformed and foreign receipts visible beyond the first 1,000 rows. - Keep Slack callback, answer, receipt and ACK rules in their provider owners. Support body-bound Cloud idle capabilities when a complete catalog exceeds the header budget. - Add focused regression coverage, an enabled quiet/reactivation recipe and explicit fixture port checks. - Keep SDK evidence tests tied to a normally built reviewed reference, with a separate strict refusal for actual unreviewed CI outputs. ## Verification App head: `6284d6881b5dc38a9ee149bcbde35e69a67dd8ca`, including actual master `dff8bed007e01c32f3826144b893ec80603d1335`. Companion Cloud head: `f83b38187dc7e6b2e219fe865dc16d268cc66fdb`, including actual Cloud master `b1dcc3421a91efffee631a61536cf48f357c22fc`. Cloud #1250 is merged at `1d07d81f23d4627ab7b6dcf49176b4128a2f8f9b` after exact-head passing checks and complete Greptile 5/5. Current exact-head CI typecheck, build, tests and aggregate verification are green; the complete edited Greptile review is 5/5 and all 13 review threads and nested comment pages are resolved. - Both latest enrollment findings were reproduced against the preceding product. The pure readiness control returned no wait for the first enrollment. The real-PostgreSQL control observed a provider call while that enrollment was still staged. - The repaired source passes three whole files: 21 readiness controls, 80 real-PostgreSQL milestone controls and 37 tenant-actor middleware controls. All 138 pass with zero excluded or TODO cases. The SQL case reconstructs the worker through seven waits, preserves the attempt budget, recovers a controlled lost enrollment reply, and commits one provider-port call and one exact grant. Replaying the selected attempt does not duplicate them. Authorization, SDK and enrollment-response ports are controlled in that SQL case. - The same files cover ordinary five-attempt exhaustion, accepted and possible sends, stale ownership, malformed bindings, attention states and current actor restrictions. One synthetic mismatched-binding parameter uses generation 2 against current generation 1; it is not evidence for an older-generation control. - The normal server-package typecheck passes at the measured source. It runs the normal Runner TypeScript, protocol, SDK and declaration prerequisites and all seven main App programs. The separate declaration configuration is also recorded. A worker union-narrowing error was found, fixed with an explicit callable/null branch, and verified by that normal compiler. No cast or test relaxation was used. - The normal merge of current master preserves all 78 owned paths and 15,783 unrelated master entries and modes. The three measured test files and their changed feature paths are unchanged. New upstream DB code adds only the normal nonunique concurrent native-session index and its migration classification. Shared and adapter-utils changes alter unrelated prompt or denial wording. The 138-case proof carries for those unchanged behaviors; it is not reported as a rerun after the merge. The normal Shared/SDK ensure, DB build, adapter-utils build and complete server-package check also pass at the current merged head. Their fresh inventory contains 8,143 outputs; canonical SQL and journal copies match. This build did not apply the new index migration. - Earlier real-PostgreSQL retained-history coverage reproduced the missing proof with 1,005 settled rows. Its repair retains the existing two-second/64-query inspection budget, current endpoint bound and exact owner snapshot. The 46 source-bound controls cover unfinished and invalid tails, exact cross-page action references, Retry, rollback and lost receipt responses. - Earlier enabled quiet/reactivation coverage produced ordinary useful work, kept configuration enabled, accepted fresh signed work plus a duplicate, and produced exactly one second useful reply. Its two App-worker windows lasted at least 30 seconds with zero measured PG writes, connection or outbound activity after settlement. That evidence stays tied to its measured revision. A missing worker EXIT marker limits that earlier proof; parent evidence does not replace it. - Earlier three-second callback, terminal ACK, current-question race and complete body-size controls remain source-bound. The 500-owner/1,000-binding body measured 313,875 bytes with a 618-byte JWT; signature tampering failed closed. - Final managed-provider acceptance now passes all five fresh Node boots: A/A, A/AB, B/A rejection, B/AB recovery, B/B final. Two independently configured, challenged, verified, linked and enrolled Slack identities exercise ACK and retirement debt separately. Normal provider factory/materialization delivers the configuration through a controlled external stack-provider boundary. The registered opening returns HTTP 200 in 21 ms within the original 3-second deadline. Two answers persist, normal ACK is accepted, exact retirement and lost acceptance/GC replies recover, and the original canonical snapshot remains unchanged without duplicate provider sends or submissions. - The separate complete committed-preflight case passes. A committed `update_attempted` response is lost before SDK entry; normal service-owner reconstruction after the original lease/deadline recovers the same useful update and one answer without a second trigger. This reconstructs service owners within one App process; the separate five fresh Node boots cover process restart and managed signing-key rotation. - Assertion-time PG socket reports balance 15 opens/15 closes across the five boots and 5/5 in preflight, peak 2 and zero live sockets or unexpected attempts. Worker terminal EXIT markers were missing and are not replaced by parent EXIT markers. These are domain/recovery and assertion-time connection proofs, not a new whole-worker terminal quiet measurement. All known owners/groups closed; independent SQL audits were empty; only successful fixture databases were removed. Earlier failed fixture databases/logs remain retained. No product guard, deadline or assertion was weakened. Exact runtime source and normal outputs are unchanged by the three test files in the current follow-up, so these source-bound proofs carry without repeating those journeys. - The preceding App CI ran 11,312 support cases: 11,289 passed, three SDK-evidence cases failed, and 20 external cases skipped. Normal CI dependency setup resolved six stale Slack patch-hash references and produced an unreviewed lock. The public SDK proof correctly refused that lock; the tests incorrectly assumed the CI output was a reviewed reference. The three-file test repair keeps the protected lock policy, source/compiler/parser/Zod pins, scanner and redaction rules unchanged. It builds an owned reviewed reference using the existing normal Shared/SDK producers for the positive seal test, while a separate negative requires actual unreviewed-build privacy refusal and no retained SDK bytes. It does not relabel the CI build as certified or admit a new lock hash. - All four whole SDK support files pass locally: public type 30, evidence 16, public-input synchronizer 8 and lock preparation 10, totaling 64 with zero excluded/TODO cases in 28.73 seconds. The genuine fresh reference build/positive seal and separate invalidated-reference privacy refusal passed within their original 120-second bounds. Original installed dependencies, SDK outputs/cache and runtime source stayed unchanged; normal global setup produced only its own Runner JavaScript outputs. PostgreSQL was denied throughout, all four worker markers were observed, and the owned process group/private test root closed. The local canonical-lock negative mutates only its own reference; fresh remote CI also passes the actual resolved-CI-lock branch: all 16 evidence and 30 public-type tests pass, including the genuine reviewed-reference positive and the strict actual-unreviewed-build refusal. The upstream synchronizer file runs its current 10 cases in CI. - The first complete CI attempt on this head also exposed two separate 30-second timeouts: one real reporter-artifact test and the `agent-test-environment` module-loading hook. Full original logs remain preserved. Their exact test, sanitizer and evidence source blobs are unchanged from the preceding head; inspection found no deterministic setup cause. After the original run finished, one retry of only the failed jobs passed on the same synthetic merge `168851e210671ebe97e7a778e8cf891807e80502`: Runner support reports 11,304 passes and 20 external skips; serialized shard 6 reports all 395 passes, including all 67 route bodies. No source, timeout or assertion changed. This proves recovery; the original causes remain unexplained. - The final pre-merge tree audit uses actual master `17136a6c25ef6cb7c816a29140a4c48c2c75fb31` (refreshed after the earlier `85dec046` audit). The expected normal squash preserves all 81 owned paths and 15,816 unrelated master entries and modes. Only OpenAPI has edits from both sides, and its complete content composition preserves the new project route and token help alongside this PR's API entries. The shared inbound selection changes preserve the same candidate set; new concurrent indexes, Native/workspace behavior and shutdown changes remain upstream behavior. The scoped form proof uses process agents and does not qualify the new Native/sandbox branches. The latest question-schema change still accepts the explicit version-1 payload exercised here. The last master delta also changes managed AI/Native credential refresh and writeback; those branches are outside the measured process-agent form paths. Existing 8,143 local outputs remain tied to their measured revision; no new whole-tree output equality or deployed suspension is claimed. - Reproduce the public path with `doc/testing/slack-cloud-qualification.md`, the companion Cloud checkout, Node 24/22 and fresh owned fixture databases. Aggregate local `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not rerun as one command set. Current exact-head required CI and the complete Greptile 5/5 review now pass; the full edited summaries and all paginated threads were read. Local worker quiet does not prove deployed host or Neon suspension. Photon and Discord remain separate work; this PR completes the described Slack slice. ## Risks Use compatible App and Cloud readers before transferring custody. Signatures prove authenticity; current authorization and exact ownership still control execution. Preserve old targets, secrets and terminal facts until their normal owners release them. Keep the managed seal identity stable while custody remains. The required Slack adapter patch remains pinned. Upstream's concurrent index uses the normal top-level migration path. ## Model Used OpenAI `gpt-6.1-sol` implemented and tested the changes. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the managing agent's exact model ID or context-window sizes. The agents used code execution and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, its version and tool capabilities - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public related issues or PRs - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the local tests listed above and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green for the current head - [x] Greptile is 5/5 with no open findings for the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 12:17:50 -06:00
DottaandClaude Opus 5.5	1b7ec13acb	perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can run in remote sandboxes (Daytona), where each turn needs the task's workspace and a runner inside the sandbox > - With a default environment, every remote turn got a brand-new sandbox: the whole workspace was uploaded again and a new runner started, even for a follow-up comment > - In a paperclip-sized workspace that made a follow-up take 30 s to first token, against 8–10 s with a reusable warm sandbox > - This pull request makes an agent that never chose a lifecycle run warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude), so a follow-up keeps its sandbox, runner and workspace > - The default applies only where warm can work: a sandbox provider that keeps reusable leases, and a managed AI account that supports a warm session. Everywhere else the run is per-turn, exactly as before > - The benefit is fast remote follow-ups by default, without re-uploading a large repository every turn ## Linked Issues or Issue Description What happened? On Daytona, with a fresh paperclip clone as the project workspace, a follow-up comment took 30 s to its first token. The run log showed `environment: fresh/created` on every follow-up. It spent 13.3 s in `workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s in `runner.session.resume`. Steps to reproduce 1. Create a Daytona environment with the defaults: lifecycle inherited, lease reuse off. 2. Run a Codex task in a large repository, then post a follow-up. 3. The follow-up gets a new sandbox, uploads the whole workspace again and starts a new runner. Expected behavior A follow-up on the same task reuses the live sandbox, its runner and its workspace (plan rule 2: no full re-upload per turn). ## What Changed - One rule for the default. `defaultsRemoteSandboxRunToWarm` (`native-runtime/native-warm-capability.ts`) decides the remote-sandbox warm default for both the lease (environment view) and the runner (native session policy), in one vocabulary (runtime backend plus ACPX agent). It is applied only when no lifecycle was chosen anywhere: admitted session, environment `runnerLifecycleMode`, or agent `lifecycleMode`. It holds only when: 1. the harness is measured (Codex, OpenCode, ACPX Claude); 2. the sandbox provider keeps reusable leases (Daytona, CreateOS and boxd declare them; E2B, Modal and the built-in `fake` do not); and 3. with a managed AI account, that harness supports a managed warm session (`supportsManagedNativeWarmSession`: Codex). - The harness resolves as the runtime resolves it. `resolveWarmProviderSelection` uses `resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is Codex and ACPX without an agent is Claude, in both decisions. A config the runtime would refuse gets no default. - Lease side (`runner-environment-lifecycle.ts`, `environment-run-orchestrator.ts`): for an eligible run only, the orchestrator asks the provider before acquisition (`providerSupportsReusableLeases` in `environment-runtime.ts`: the acquisition rule, one registry lookup for a plugin provider, no worker-ready wait). With support, the run-scoped view sets `reuseLease: true` and forwards the warm idle window to the provider's auto-stop. The shared environment is never rewritten. Every other run resolves exactly as on master, without the probe. - Runner side (`native-lifecycle.ts`, `runtime-selection.ts`): the native policy applies the same rule to the acquired lease (`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If acquisition downgrades the lease anyway, the run falls back to per-turn; a default never produces `runner_warm_environment_requires_reusable_lease`. - Idle window. The remote default uses the window the environment view forwards to the provider: an inheriting environment's `runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes. The sandbox execution target now carries the environment's window (`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution target) so the runner and the provider auto-stop agree. An explicit warm choice and the local default are unchanged. - Unchanged: explicit agent choices, environments that pin a lifecycle, plugin-task environments, admitted sessions (`persistedPolicy`), and local behavior. - `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit --lifecycle default` sandboxes as retained, because they now default to warm. - `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the ranked waits, and the gate. ## Verification - Unit and integration tests, one per rule: - Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease → per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases → environment unchanged, no warm window), `environment-runtime-reusable-lease-probe.test.ts` (`fake`, an undeclared plugin, a declared plugin whose worker does not verify the reuse methods, and a plugin without a running worker read false), `runtime-selection.test.ts` ("defaults an agent without a lifecycle on ...": reusable → warm/keep_running; reuse configured on an unsupported provider → per_turn/destroy_after_turn; ephemeral → per_turn), `environment-run-orchestrator.test.ts` (probe result decides `reuseLease` and the forwarded window). - Rule 2 (explicit configuration wins): explicit per-turn agent, pinned environment, admitted session, configured `idleTimeoutMs`, managed Claude per-turn, managed Codex warm; the orchestrator does not probe when a lifecycle is explicit. - Rule 3 (omitted provider): `{}` resolves to Codex in the environment view; `runtime-selection.test.ts` "agrees with the environment view's lease for ..." feeds the environment view's lease into selection for Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without an agent, and Copilot (per-turn), and checks both decisions agree. A plugin-task target stays out of the default. - The two suites that failed on the previous head (`runtime-selection.test.ts`, `workspace-recovery-preparation-dispatch.test.ts`) pass locally. Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a nested worktree; they do not touch this change and pass in CI. - End-to-end check of this default on Daytona (before the gate was added; Daytona keeps reusable leases, so the gate does not change this path): a user-default environment (lifecycle inherited, lease reuse unset, `--daytona-inherit`), a fresh paperclip clone, built from the image's source revision with this change (branch measure/remote-warm). \| \| Before (per-turn sandbox) \| After (this change) \| \|---\|---\|---\| \| Follow-up, Codex \| 30.4 s \| 7.6 s \| \| Follow-up, Claude \| about 30 s \| 11.0 s \| \| First follow-up after an idle K1, Codex \| 45.9 s \| 20.3 s \| \| First follow-up after an idle K1, Claude \| 38.3 s \| 18.3 s \| The run-log reuse records show the follow-ups reusing the workspace, environment, runner and provider session, with no `workspace_sync.transfer`. First turns still upload the workspace once (14.8 s and 26.1 s). Every sandbox was cleaned up. - Daytona restart chaos wave (K1/K2/K4 with a mid-run comment, inheriting environment, agents with no lifecycle, runner image built from this branch): wave `w-20261110T171438-s20048`, seed 20048, on `b7f6243d60` with image `paperclip-daytona-runner@sha256:76a33275…6fe29` (built from this branch by the `image_only` runner workflow; runner content id `ed503454…a479` matches every later commit here, which change only server, adapter-utils types and tests). `--daytona-inherit --lifecycle default --adapters codex,claude --workloads A1,C1 --triggers time,comment --faults K1,K2,K4 --tasks 6 --concurrency 3`. - 6/6 tasks pass (R1–R6 and T), all 6 crossed a restart; cells: Codex A1 K2, Codex A1 K4, Claude A1 K4, Claude C1 K1, Claude C1 K4, Codex C1 K2 at the mid-run comment. No task needed Retry (R3 n/a on every task; one Claude task recovered automatically from `process_lost`). No open signatures, 0 speed breaches. - The default applied: the R6 sweep found 6 retained reusable leases (one per task, Daytona keeps them) and no orphan or leaked sandbox; the environment was deleted cleanly. - Timings (Daytona, first turns, n=3 each): TTFT p50 Codex 21.9 s, Claude 31.0 s. Restart recovery p50/p95: Codex 10.5 s / 29.6 s (n=3), Claude 8.8 s / 23.9 s (n=2). Warm follow-up TTFT has no clean sample in this wave (both follow-ups were crossed by a fault), so the follow-up comparison above (30.4 s → 7.6 s Codex, ~30 s → 11.0 s Claude) remains the speed evidence; no separate per-turn chaos wave was run. - An earlier attempt, `w-20261110T170357-s20048`, never started tasks: the first sandbox from the new image stayed `creating` past the harness's 300 s Codex qualification bound (Daytona building the new snapshot). It left one sandbox that the harness reported after deleting the environment; the rerun on the cached snapshot is the wave above. - The idle-window fix (`9a74116082`) landed after the wave; it changes only runs whose inheriting environment sets `runnerIdleTimeoutMs`, which this wave's environment does not. ## Previous-release story No persisted state changes shape. A session admitted per-turn before this change keeps its saved policy (`persistedPolicy` wins) until a new session starts. A run in flight at the cutover keeps its admitted lifecycle for recovery (`admittedLifecycleMode` wins in the environment view). ## Cost and opt-out - Cost: after a turn, the sandbox and its warm runner stay up for the warm idle window (the environment's `runnerIdleTimeoutMs`, else the agent's `idleTimeoutMs`, else 5 minutes). Then the controller suspends the runner, and the provider auto-stop follows the same window. One idle window per active task. - Opt-out: set the agent's lifecycle to "Turn by turn" (`lifecycleMode: per_turn`), or set the environment's `runnerLifecycleMode` to `per_turn`. Either wins over this default. ## Reuse scope, restart and saved work - One lineage per sandbox. The Daytona plugin keys a reusable lease by the agent plus the execution workspace (or issue). Another task never gets a sandbox holding a different task's files, and subtasks still get fresh workspaces from the project base, so they get their own sandbox. The workspace is adopted only after its stamp matches the host's. - After a restart, a warm sandbox is reattached or replaced through the existing warm recovery and owner fencing. A sandbox with a live owner is never claimed twice. The Daytona chaos wave above checks this. - No lost work. Each turn's workspace copyback (`workspace_finalize`) runs at the end of the turn, before any idle suspend, as for per-turn runs. ## Risks - Cost: an idle sandbox stays up for the warm idle window (5 minutes) after a turn, then suspends. Only on providers with reusable leases. - Probe cost: one plugin-registry lookup before acquisition, only for runs the default can apply to. No wait for a worker: a provider whose worker is not running yet reads as unsupported, so that run is per-turn. - Known gap (pre-existing, not widened): with an explicit agent warm choice on an inheriting environment that sets `runnerIdleTimeoutMs`, the provider gets the environment's window while the host uses the agent's, as on master. - Restart behavior on warm remote sandboxes uses the existing warm recovery paths. The Daytona chaos wave above checks K1, K2 and K4. - Sessions admitted per-turn before this change stay per-turn until a new session starts. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution, Daytona measurements). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 11:49:14 -06:00
Nicky LeachandPaperclip	98ce93dd4e	Choose project storage from compatible environments (#20587 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects supply repository files to agent runs through core staging or a storage plugin. > - A storage plugin's default claim silently attached new projects even when their agents used a different environment. > - An environment name or driver type does not establish shared storage access. The existing binding and task-driver contracts do. > - This pull request groups creation choices by that contract and selects storage from the creator's effective environment. > - The benefit is a visible, validated choice and a clear error before an incompatible managed workspace operation. ## Linked Issues or Issue Description Fixes #20459. Related: #20156 covers a separate provider-failure diagnostic. The PR search found no other fix for #20459. ROADMAP.md has no duplicate project-creation change. ## What Changed - Show Available in when configured environments require different project storage. Each option names the environments that can use the same storage owner. Hide the selector when all choices are compatible. - Derive compatibility from repository staging and managed binding ownership. Local, SSH, and sandbox environments use core staging. Managed storage requires a registered task driver from the same installed storage plugin. Separate storage plugins remain incompatible. Unknown plugin contracts and the fake sandbox are excluded. - Follow the creating agent's environment, or the person's primary agent, then the instance default and local fallback. Apply managed-only and forced Kubernetes policy. The product has no separate user or company environment-default field today. - Add a company-authorized discovery endpoint and an optional creation `environmentId`. Validate the choice again on save. Store the storage owner without overriding agent run environments. Preserve explicit API storage choices and existing project ownership. - Keep the hidden repository-free private-task project independent of environment availability. Submit the observed storage owner as `expectedStorageProviderPluginId` so a changed contract fails at save without setting a workspace policy. Instance task-isolation defaults stay inherited. - Reject an incompatible managed workspace before a provider operation. Name the storage and selected environment in the error. Add regression tests, API documentation, creation-story fixtures, and minimal corrections to obsolete map text. ## Verification - 102 tests passed across 7 files: project environment integration, installed storage policy, repository persistence, project environment routes, project creation telemetry routes, OpenAPI, and NewProjectDialog. Run with `node node_modules/vitest/vitest.mjs run server/src/__tests__/project-environments.test.ts server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/project-repositories-persistence.test.ts server/src/__tests__/project-routes-env.test.ts server/src/__tests__/project-goal-telemetry-routes.test.ts server/src/__tests__/openapi-routes.test.ts ui/src/components/NewProjectDialog.test.tsx --silent`. - The review fixes cover both archived-default private-task cases and changed-storage rejection. With both instance isolation flags enabled, a new core repository project keeps a null policy and resolves to an isolated Git worktree. The complete affected set passes all 102 tests. - The CI query-error policy failure was fixed by using the shared query view. The form and shared query-view suites pass 24 tests, including cached choices during a transient refetch failure. `node scripts/check-query-error-rendering.mjs --enforce --summary` reports zero findings. - CI exposed stale SDK source hashes and heartbeat fixtures that paired plugin storage with Local. Refreshed exactly four shared-source hashes with the maintained generator (runtime pins unchanged). All 53 SDK privacy/source-proof tests, including real built SDK output, pass. All 7 corrected managed-storage heartbeat scenarios pass; the fixtures declare the owning plugin's task environment and substitute transport at the existing execution seam. - Shared and UI TypeScript checks pass. The UI production build passes. `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and diff checks pass. - Tested UI states: compatible choices, incompatible groups, default within a group, manual override, create failure, loading, discovery failure and retry, and an unavailable default. Tests use the production form and existing design tokens. - The integration tests use real isolated PostgreSQL on macOS arm64 with Node 26.12.0. They cover user/agent defaults, storage persistence, policy restrictions, company and authentication denials, and rejection before a provider call. Discovery uses saved configuration and makes no worker or provider requests. - Local verification limits: `pnpm -r typecheck` and `pnpm build` stop in unchanged adapter-utils code because the installed ACPX types lack `inheritProcessEnv`. A direct server check also finds an unchanged AI SDK type-version mismatch. The existing heartbeat test `inherits the instance default environment across companies while preserving explicit agent overrides` times out at line 518 on both this change and unmodified master with the same dependencies. The full `pnpm test:run` attempt passed all six chat shards (1,948 tests), then the Git workspace expiration suite reported 14 failures. I stopped that broad run. Two sampled failures reproduce on unchanged master with the same `unknown_worktree_git_files` refusal; the other expiration failures were not independently compared. A later broad workspace-preparation run also hit workspace validation failures and was stopped; the seven corrected fixture scenarios pass independently. These limits do not establish a full local suite pass. Hosted CI remains required. - Initial missing-package and loopback-denial failures were resolved by repairing isolated dependency links, building the SDK, and running database tests with loopback access. No deployed or live-provider execution is claimed. ## Risks - Creation defaults now follow the effective environment. A plugin's `defaultForProjects` claim alone no longer selects storage. This is the intended behavior change. - Compatibility covers repository project storage. It does not guarantee arbitrary host paths, retained-workspace moves, adapter/image support, credentials, or worker readiness. Matching provider configs cannot prove a cross-plugin storage contract. - A person's primary agent supplies the personal default because there is no separate personal environment setting. An unavailable configured default requires a valid explicit choice or a configuration repair. - No migration runs. Existing projects keep their storage owner. This change does not detach existing managed data or provide a storage migration. - Full local verification has the limits listed above. CI and standard Greptile review are pending. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 17:38:31 +00:00
DottaandClaude Opus 5.5	8e332ccb5e	fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 What happened On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). Expected behavior - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. Steps to reproduce 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/*` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 11:01:44 -06:00
DottaandClaude	84007892e9	refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` is the instruction file that each human and AI contributor reads before a change > - Pull request #20557 shortened the file, but several sections still held detail that applies to only some changes > - The scoped-rules section told agents to read each matching rule in full, and most changes match five rules > - The core engineering rules mixed three product invariants with process conventions > - This pull request moves that detail to the docs and to scoped rules, and adds a section on how to fix a bug > - The benefit is a 211-line file that states what applies to every change and points to the rest ## Linked Issues or Issue Description Refs #20557 Issue type Outdated or unclear contributor documentation. Where is the issue? `AGENTS.md` at the repository root, and `rules/`. What's wrong? - The scoped-rules section made agents read the full body of each matching rule before a change. The rule bodies total about 47 KB. - Sections 4, 6, 8, 9 and 10 repeated detail that belongs in `doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and `CONTRIBUTING.md`. - Section 4 had a database reset command for `data/pglite`. That directory does not exist. - Section 5 had seven rules. Four of them were process conventions or path-specific detail. - The file did not say how to fix a bug. Suggested fix Make the rule headers the index. Move section detail to the document or rule that owns it. Keep only the three invariants in section 5. Add a bug-fixing section. ## What Changed - The scoped-rules section now tells agents to list the rule headers with one `grep` command. Agents read a rule body only for a design decision in its area, when the description is not sufficient, or when a reviewer cites it. - `rules/README.md` says that each `description` must state the required behavior in one sentence. - `rules/pull-request-review.mdc` has a description that states its requirements. - Section 4 (dev setup) and section 11 (pull requests) now point to `doc/DEVELOPING.md` and `CONTRIBUTING.md`. - The schema change workflow moved from section 6 to `doc/DATABASE.md`. - The API and auth expectations moved from section 8 to the new `rules/api-route-expectations.mdc`. - Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer tells contributors to run `pnpm check:token-gates`. - Added section 10, "Fixing a Bug": reproduce the error, write a failing test, write the fix, see the test pass, and verify the fix. - Section 5 now has three rules: company scope, contract synchronization, and the control-plane invariants. - The company-scope exceptions moved to the new `rules/company-scope-exceptions.mdc`. - Removed the strategic-docs rule. The plan-doc convention moved to the repo map. The artifact rule moved to section 7. - Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now cite `rules/data-paths.mdc`. - Regenerated the SDK public source pin, because one of those comments is in `packages/shared/src/crash-points.ts`. - Added both new rules to `.greptile/files.json`. - The API route rule applies only to routes that read or write company data. Its status code list is a list of common codes. - The company-scope rule lists the sidebar company order as an instance-wide exception. - The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for the Telemetry workflow. ## Verification - Open `AGENTS.md` and confirm that each link resolves. A local script checked this. - Run `grep -E '^(description\|globs\|alwaysApply):' rules/*.mdc` and confirm that each rule prints a description. - Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`. It passes locally. - `packages/shared/src/crash-points.test.ts` passed locally before the rebase onto `master`. - Five source files changed in comments only. No other test suite ran locally. ## Risks - An agent that reads only rule descriptions can miss a detail in a rule body. Reviewers still apply the full rules. - Pull requests and the definition of done are now sections 11 and 12. Sections 1 to 9 keep their numbers. - Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`. They are historical records. - Pull request #20565 also changes `AGENTS.md`. The second one to merge needs a conflict fix in section 1 and section 2. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 11:00:31 -06:00
DottaandClaude Opus 5.5	d0c30e69a9	fix(storage): abort a snapshot upload a restart cut off once its save attempt is superseded (#20542 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote workspaces save their files as encrypted snapshots in object storage. Snapshots over 16 MiB use S3 multipart uploads. > - A server restart can cut off a snapshot upload in the middle of an `UploadPart` request. > - The server aborts a multipart upload only in-process, on an error or an abort signal. A SIGKILL can never do that. On SIGTERM, the drain step timed out while the store held the part, and the process exited without `AbortMultipartUpload`. > - The incomplete upload then stays in the bucket. Nothing records its upload id and nothing finds it again, so incomplete uploads pile up and the operator pays for their parts. > - This pull request aborts the incomplete uploads of each save attempt when that attempt can no longer win, with no polling and no schema change. > - The benefit is that no incomplete upload outlives its save attempt. ## Linked Issues or Issue Description What happened? A restart-chaos wave (Daytona, R1 workload, storage fault ST4, `--storage-trigger snapshot-part`, faults K1 and K2, master `fea4530a30`) passed all 3 tasks. All 3 committed snapshots read back intact. But the storage audit found 3 incomplete multipart uploads. Each one was the exact upload that a restart cut off in the middle of `UploadPart`. The retried save then published a different attempt of the same snapshot. Expected behavior When a save attempt can no longer become the snapshot's object, its incomplete multipart uploads are aborted. The storage audit shows 0 incomplete uploads after the wave. Steps to reproduce 1. Run a snapshot save larger than 16 MiB against S3. 2. Kill the server (SIGKILL), or send SIGTERM while the store holds an `UploadPart` request. 3. Let the save retry and publish. 4. List the bucket's multipart uploads. The cut-off attempt's upload is still there. Paperclip version or commit master `fea4530a30` and later. Relevant logs or output `shutdown step timed out; continuing shutdown without it {"step":"delivery_work_stop","durationMs":20001}`, then `Paperclip did not stop within 30000ms; sending SIGKILL`. ## What Changed - The S3 storage provider and the storage service get `abortIncompleteUploads(objectKey)`. It lists the multipart uploads under that exact key (`ListMultipartUploads` with the key as the prefix), skips other keys that only share the prefix, and aborts each one. `NoSuchUpload` counts as already done. The local-disk provider has no multipart uploads and does nothing. - Save publication: the save journals every attempt's object key in `metadata.uploads` before any remote I/O. When one attempt moves the snapshot from `pending` to `uploaded`, no other attempt can win any more. The winning process now aborts the incomplete uploads of every other journaled attempt key. Each abort is best effort and bounded to 5 s. A snapshot with one attempt sends no extra request. - A losing uploader that is still live: if its upload fails after another attempt has published the snapshot, it now returns the published snapshot (after it verifies it) instead of the error. So an abort from the winner never fails a save that is still running in another process. - Snapshot retention: before it deletes a retired snapshot's objects (this includes saves that #20502 retires as abandoned), retention aborts the incomplete uploads of each attempt key. A store that refuses the listing (for example a missing `s3:ListBucketMultipartUploads` permission) logs a warning and does not block the deletion. - Abandoned saves: when the ownership reconciler retires an abandoned save (`remote_snapshot_save` and `remote_snapshot_ledger` settlement, from #20502), it now aborts the incomplete uploads of every journaled attempt key after its transaction commits. The owner run is terminal and the save work is settled, so no live uploader can own those uploads. This is the durable path for SIGKILL: the reconciler pass after boot settles the dead save and cleans its uploads. - SIGTERM drain: the multipart upload code keeps an in-process set of running uploads. Shutdown aborts them right away when `delivery_work_stop` times out. It also aborts them after the heartbeat run drain, before the finalizer and workspace-recovery waits, after a 5 s grace for them to finish. It waits at most 10 s for the `AbortMultipartUpload` calls. Shutdown does nothing extra when no upload runs. - No polling and no periodic sweep. All triggers are events that already exist: publication, abandonment settlement, retention of a retired snapshot, and shutdown. Previous-release story (`rules/upgrade-safe-run-state.mdc`): this adds no persisted state and changes no saved shape. A snapshot that the previous release saved with cut-off attempts in `metadata.uploads` carries forward. Its incomplete uploads are aborted when the new release publishes a retry of that snapshot, or when retention retires the snapshot. ## Verification - Red before, green after (all fail on master, pass with this change): - `server/src/services/remote-workspace-snapshot-storage.test.ts`: "aborts the incomplete upload of an attempt a restart cut off once a retried save publishes" and "adopts the published save when another uploader aborts this attempt's upload". - `server/src/services/remote-workspace-snapshot-retention.test.ts`: "aborts each attempt's incomplete multipart upload before deleting a retired snapshot's objects". A second test proves that a refused listing does not block retention. - `server/src/__tests__/storage-s3-multipart-shutdown.test.ts`: file and streamed uploads whose part the store holds are aborted at shutdown, and the provider aborts only the exact key across listing pages. - `server/src/__tests__/remote-snapshot-ledger-settlement.test.ts`: "aborts the incomplete multipart uploads of every attempt when the reconciler retires an abandoned save" (both settlement kinds; a key outside the snapshot's prefix is skipped). - `tests/restart-chaos/storage-orphan-uploads.test.ts` (MinIO): the real provider aborts both cut-off uploads of one attempt key, and keeps a sibling attempt and a longer key that shares the prefix. - `cd server && npx vitest run src/__tests__/storage-s3 src/storage src/__tests__/remote-snapshot-ledger-settlement.test.ts src/services/remote-workspace-snapshot src/services/remote-workspace-transport.test.ts src/__tests__/remote-workspace-file-resources.test.ts`: 19 files, 517 tests pass; with `src/services/recovery`, `src/__tests__/ownership` and `heartbeat-workspace-busy.test.ts`, 32 files and 580 tests pass. - `cd server && npx tsc --noEmit` and `node scripts/typecheck-test-projects.mjs tests/restart-chaos` pass. - Live proof, Daytona R1, ST4, `--storage-trigger snapshot-part`, K1/K2, seed 1033, the evidence wave's flags, image `paperclip-daytona-runner@sha256:05a4a092…` (controllerMatch: match): - Before (master `fea4530a30`): `w-20261110T130541-s1033`: 3/3 pass, 3 incomplete multipart uploads. - `w-20261110T134623-s1033` on an earlier revision of this PR (publish + retention hooks only): 2 incomplete uploads remained. Those saves were abandoned, not retried, so neither hook ran. This led to the settlement hook. - After (head `8bc8fc66d0`): `w-20261110T141144-s1033`: 3/3 tasks pass, exit 0, every fired restart proved it cut the held `UploadPart`, the reconciler settled 3 abandoned saves, shutdown logged one in-flight abort, and the storage audit shows 0 incomplete multipart uploads* and 3 committed snapshots that read back intact. ## Risks - A live uploader in another process can lose its incomplete upload when a different attempt publishes first. That uploader now adopts the published snapshot, so its save still succeeds. - Narrow gap: an uploader that recorded its attempt but had not yet sent `CreateMultipartUpload` when the winner published, and that then crashes during its upload, leaves an incomplete upload until retention retires the snapshot. An S3 lifecycle rule for incomplete uploads stays a good backstop. - Stores without `ListMultipartUploads` permission log a warning and keep the previous behavior. - Shutdown can take up to 10 s longer, and only when an upload is still running after the drain. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 10:56:52 -06:00
DottaandClaude Opus 5.5	10bc6431da	test(restart-chaos): measure TTFT in a real repository workspace, after an idle restart, and on Daytona (#20047 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos TTFT driver measures time to first token, but only with a tiny fixture workspace and only locally > - Dotta asked whether speed is good with a workspace like the paperclip repository itself, and that had never been measured > - This pull request lets the driver run in a real repository workspace, measure the first follow-up after an idle restart, run on Daytona (per-turn or warm reusable sandboxes), and report each run's workspace operations > - The benefit is a repeatable large-workspace speed measurement; its first run found a full workspace re-upload on every remote turn ## Linked Issues or Issue Description What happened? The TTFT driver seeded only a tiny fixture repository, created tasks outside any project, ran only locally, and could not show workspace operation durations. So large-workspace speed (worktree creation, provision, remote upload) was never measured. Steps to reproduce 1. Run `ttft-baseline.ts` with any options. 2. Every task runs without a project workspace and on the local host. 3. Nothing in the report shows worktree, provision or sync cost. Expected behavior The driver can measure a real repository workspace, locally and on Daytona, and names the workspace operations each run waited on. ## What Changed - `--workspace-repo <path>`: the seed uses that repository as the project's primary workspace (git_worktree isolation), and tasks are created in the project. - A `postfault` phase: K1 while idle, then a follow-up on each task; reported in its own table. - `--env daytona`, `--daytona-image`, `--daytona-cpu`, `--daytona-warm` (the reusable warm Daytona fixture). The Daytona controller preparation moved to `daytona-controller.ts`, and every sandbox is cleaned up at the end; incomplete cleanup fails the run. - Each run's workspace operations (`/api/heartbeat-runs/:id/workspace-operations`) are saved, and printed ranked by duration. ## Verification - `tsc -p tests/restart-chaos`: no errors in harness files. - Used for the large-workspace measurement, a fresh paperclip clone on local and Daytona (per-turn and warm). First turn / follow-up: local 31–34 s / 7 s; Daytona per-turn 58–63 s / 30 s; Daytona warm 45–51 s / 8–10 s. Every Daytona sandbox was cleaned up (environment deleted, none remaining). ## Risks - Test harness only; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution, Daytona measurements). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 10:00:45 -06:00
DottaandPaperclip	ce1ee9e89f	fix: stop prompts from encouraging subtasks (#20362 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read runtime prompts, tool help, skills, and role instructions. > - Several of these texts tell agents to create child tasks as a normal next step. > - One accepted-plan prompt even forbids work on the current task. > - Task parentage is useful, but it does not need a prompt that encourages task creation. > - This pull request removes those prompts and keeps the task APIs available. ## Linked Issues or Issue Description Refs #10642. This change adjusts prompt wording; it does not add delegation-cycle guards. Current behavior Prompts tell agents to create children after plan approval, use children for long work, and create a task just to pass context. Onboarding requires a child even for a single request. Proposed behavior Continue work on the current task. Keep explicit assignments, optional task creation, parentage, dependencies, and approval gates. Use the separate-task recipes when the user requests that structure or when continuing existing work. Reason and benefit Reduce needless task trees without removing subtask support. Keep the patch limited to wording, existing test expectations, and generated catalog data. ## What Changed - Align accepted-plan prompts and remove direct child-creation advice from runtime and error text. - Make task tool descriptions neutral. Keep their fields, defaults, and handlers. - Update the main skill, onboarding, CEO instructions, company templates, specialist workflows, and matching guides. - Keep historical release snapshots intact. Refresh the explicitly seeded schema fixture's catalog digest; the server still verifies historical release snapshot hashes. - Refresh the affected generated catalogs, capability inventory, semantic contracts, and fixture manifest. ## Verification - Final tested head: `c6e304d0c105cdbfbe0b2ab0a233307fcd91a04b`. - All 66 check runs are complete: 60 pass, one is neutral, and five are skipped by CI conditions. Snyk also passes. No checks fail or remain pending. - [PR CI](https://github.com/paperclipai/paperclip/actions/runs/42846226554) passes typecheck, build, server/general tests, all eight E2E shards, Runner support, Runner Rust, and the aggregate verify gate. [Windows](https://github.com/paperclipai/paperclip/actions/runs/42846226546) and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42846226108) also pass. - [Greptile](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751419835) and [Paperclip review](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751454010) give this exact head 5/5. All six inline threads are resolved. Neither current summary has findings. - Merged master `bf0220d34a156c81853755c3d1365ddc527037de`. Resolved three generated-file conflicts with the canonical generators. - Local verification passes: 473 focused prompt, continuation, UI, catalog, runtime and native-execution tests; Runner TypeScript build; generated metadata; UI token gates; and diff checks. - The prior head `c22018cb0f` failed three SDK evidence tests because two source hashes were stale. Reproduced all three failures locally. The existing generator refreshed only the hashes for the denial-copy file and its test. All 53 affected evidence/source-pin tests then passed, including private-value rejection. No scanner logic, assertions, or runtime dependency pins changed. Current CI confirms the support suite passes. - A prior commit failed the Rust process reaping test. Its cause remains unproven. No Rust code or assertions changed. Later Rust runs pass; this does not establish the cause of the earlier failure. - Local preflight built shared/plugin SDK packages, then stopped because `cargo` is absent. Current-head CI supplies Rust and full build coverage. `pnpm verify --explain` selects broad required CI; full local suites were not repeated. ## Risks - Wording changes guide the model; they do not enforce a ban on subtasks. - Existing installed skill versions and saved custom instructions need their normal refresh process. This PR does not migrate company configuration. - Existing approved task assignments and approval gates remain in effect. ## Model Used - OpenAI GPT-6, through Codex, with reasoning, tool use, and code execution. The exact routed model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 09:58:54 -06:00
DottaandClaude Opus 5.5	d0057ee384	test(restart-chaos): fire a planned restart at native start (#20538 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) restarts the server during real agent runs and grades the recovery. > - Several native recovery fixes only apply when a restart lands after the run has started its native runner. Examples: a restart claim whose lease lapses in a storage outage (#20529), and a K2 seconds after native start on a remote sandbox. > - On Daytona R1, workspace preparation takes 100-250 s and varies a lot under storage faults. A time-based fault and `--storage-trigger run-log` both land while the run is still preparing. In three waves, none reached native start. > - This pull request adds `--native-trigger`, which fires the planned fault a set delay after the run's own `native.process_start_requested` event or first provider event. > - The benefit is that a wave can reliably restart the server at native start and prove those recovery paths live. ## Linked Issues or Issue Description Refs #20529, Refs #20514 What happened? Waves `w-20261110T113240-s11936`, `w-20261110T120901-s11937` and `w-20261110T124555-s11936` (Daytona R1, K2) were meant to restart the server after native start. Every K2 landed during remote workspace preparation, before native runtime selection. The drain took the legacy interrupt path, so the native recovery path under test never ran. Expected behavior A wave can fire its planned fault a known number of seconds after the run starts its native runner. Steps to reproduce 1. Run a Daytona R1 wave with `--faults K2 --fault-after-seconds 110`, optionally with `--storage-faults ST1`. 2. Read `server.log`. The graceful drain lists the runs under `interruptedRunIds`, not `restartSuspendedRunIds`. Paperclip version or commit `master` at `adb0e77a4c`. ## What Changed - New `tests/restart-chaos/native-trigger.ts`: `--native-trigger native-start\|provider-event[+<seconds>]` parsing and a `NativeTrigger`. The trigger arms each task's planned fault at its planned time. Each wave loop pass, it reads the task's running runs' events through the board API with a per-run cursor. It fires the fault the given delay after the first matching event created after arming. It records receipts, misses (`task_rested`, `no_matching_event`), the fault id, a summary section and a shortfall check. - `config.ts`, `types.ts`, `main.ts`: registry lines only. They add the option, construct the trigger next to `--storage-trigger`, arm faults instead of firing them by time, tick the trigger in the wave loop, keep the loop alive while it fires, and write `native-trigger.json` plus the summary section. - New `tests/restart-chaos/native-trigger.test.ts`: parsing and conflicts, event matching, firing after the delay, ignoring events from before arming and from non-running runs, waiting while another fault runs, misses, and fault linking. - `tests/restart-chaos/README.md`: a "Restart at native start" section. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/native-trigger.test.ts tests/restart-chaos/storage-trigger.test.ts`: 16 passed. The new tests fail on `master`, where the option and module do not exist. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Test-only harness change; no product code changes. A live wave with this trigger follows as the gate wave for #20529. ## Risks - Low risk. This is test harness only. A wave without `--native-trigger` takes no new code path except a no-op `tick` call. - The trigger polls the board API once per wave loop pass (1 s) for at most the armed tasks' running runs. That adds a little load on the server under test. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 09:08:34 -06:00
DottaandClaude	8d2bf78323	test(runner-e2e): resync the SDK public source pins with master (#20550 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite checks that the plugin SDK's public types come only from reviewed source inputs, pinned by git blob hash in `tests/runner-e2e/source-sdk-public-inputs.json` > - #19614 landed those pins, generated against an older master; since then `package.json` and ten files under `packages/shared/src` changed > - So `source-sdk-public-type.test.ts` (2 tests) and `source-sdk-evidence.test.ts` (1 test) fail the `ci / Runner support suite` check on every PR merge ref > - This pull request regenerates the source pins with the script made for that job > - The benefit is a green Runner support suite for every open PR again ## Linked Issues or Issue Description What happened? Since #19614 merged, `ci / Runner support suite` fails on every PR (for example #20047 and #20538) with `SDK public source proof unavailable` and `Source SDK evidence privacy check failed`. Steps to reproduce 1. On master at `902f376796`, run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk`. 2. Two tests in `source-sdk-public-type.test.ts` fail. Expected behavior The source pins match the committed tree, and the suite passes. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.json`: source pins regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs `902f376796` --write` (11 changed blobs, plus new source files). The script reports `runtimePinsChanged: false`: the lock, compiler, parser and zod pins are unchanged. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk`: 5 files, 68 passed, 1 skipped (before: 2 failed). ## Risks - Test registry only, no product code. The next change to a pinned input under `packages/shared/src` or `packages/plugins/sdk/src` will need the same resync, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 08:33:32 -06:00
Dotta Claude Opus 5.5 Paperclip	f4337bb5fb	test(evals): land the chat catalog eval harness on master (#19614 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 What existing behavior does this improve? The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. Subsystem affected `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. Proposed change Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 08:02:11 -06:00
DottaandClaude Opus 5.5	7329eaf353	perf(ui): open sidebar Recents tasks from cache (#20450 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI sidebar has a Recents list of the tasks a person opened last > - The Recents list renders from a local snapshot, but its links had no warm-up. A click waited for the task-detail code and about nine first-frame reads > - Task lists (Inbox, Tasks) already warm a task on hover, press and click, and #20355 added a batched `chat-bootstrap` read that had no UI consumer > - The hosted database is CPU-bound, so a fix must add at most one read per page load and must not compete with the page's own reads > - This pull request gives Recents the same open-intent prefetch as task lists, warms the task-detail code, and preloads the top ten recent tasks with one `chat-bootstrap` request after the page is idle > - The benefit is that a click on a recent task shows its chat from cache, without a wait for the API ## Linked Issues or Issue Description Refs #20355 (server read this PR consumes). Builds on #20389, #20398 and #20419. Is your feature request related to a problem? Opening a task from the sidebar's Recents was slower than opening it from a task list. The Recents links had no prefetch. The task page waited for its first-frame reads after the click. Describe the solution you'd like Recents links use the task lists' open-intent prefetch. When the page is idle, one batched request loads the first frame of the most recent tasks into the task page's caches. Describe alternatives you've considered Per-task prefetch of every recent task. That makes up to ten times nine requests per page load, which the hosted database cannot afford. Additional context I searched open PRs and remote branches for "chat-bootstrap" and "instant task opens". The only related work is the server-side branches that merged as #20355. There is no other in-flight UI consumer of `chat-bootstrap`. ## What Changed - `ui/src/hooks/useIssueNavigationPrefetch.ts`: the open-intent handlers from `IssueLinkQuicklook` move into one hook (light warm on pass-over; `history: true` on a resting hover or focus, a mouse or pen press, and a click). `IssueLinkQuicklook` now uses the hook. Its behavior does not change. - `ui/src/components/RecentTaskNavItem.tsx`: one Recents link wrapper that applies the hook. The classic list (`SidebarRecentTasks`), the Coordinator task rows and `DesignSidebar` use it. `SidebarNavItem` accepts the handlers through a new optional `linkHandlers` prop. - Classic Recents links now open `/issues/<identifier>` (Coordinator already did). The old UUID link remounted the task page after a redirect to the identifier route. - `ui/src/hooks/useRecentTaskOpenWarmup.ts`: when Recents shows a task, it fires the task-detail code warm-up (`useWarmTaskDetailCode`). After two frames, a visible tab, no fetching queries and an idle callback, it sends one `issues/chat-bootstrap?ids=` request for the ten most recent tasks. It skips tasks whose first frame is cached and fresh. It sends no request for an empty list or a hidden tab. It sends at most one request per account per page load. Unmount or an account change aborts the request. - `ui/src/lib/issueChatBootstrapCache.ts`: `seedIssueChatBootstrap` writes each field under the keys the task page reads, for the requested ref, the id and the identifier: core issue, recent comments, comment input facts (one batch, through the new `seedIssueIdBatch`), recent activity, the recent run window (through the new shared `buildRecentIssueRunWindow`), interactions, attachments, live runs and active run. An explicit `null` active run is stored. A missing field is not stored. It does not replace data that is as new or newer, data with a read in flight, or data with a pending invalidation. Keys whose page merges observed history (comments, activity, runs, input facts) are filled only when empty. Items for another company are ignored. - The seed fills the `?view=core` key, not the full detail key. `seedIssueDetailCache` is not used: the core payload cannot stand in for the detail read's related data, and seeding it there would suppress that read. - `issuesApi.chatBootstrap` client method. - `tests/perf/collections/recent-task-open.spec.ts` and its README section. ## Verification Local disposable instance (`local_trusted`, production UI build, embedded PostgreSQL, collection fixture plus 12 comments on each of 8 recent tasks, streamlined UI with the Coordinator sidebar). Journey: open Projects with 8 Recents, wait for network idle plus 3 s, click a recent task, measure until its latest comment is on screen. Each run uses a fresh browser context (empty HTTP cache and storage apart from Recents). Median of 5 runs, same server and fixture for both arms: \| Arm \| Click to first chat frame \| Idle page API requests \| \| --- \| --- \| --- \| \| `master` (`b6f4008f9b`), loopback \| 227 ms (first run: 249 ms) \| 24 (range 21–25) \| \| This PR, loopback \| 134 ms \| 22 (range 21–27) \| \| `master`, +150 ms on each API response \| 455 ms \| 19 \| \| This PR, +150 ms on each API response \| 135 ms \| 20 (+1 `chat-bootstrap`) \| With added latency, the first frame no longer depends on API latency. - `PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/collections/playwright.config.ts recent-task-open.spec.ts recent-task-summaries.spec.ts`: 5 passed. On `master` the two assertion cases fail (0 `chat-bootstrap` requests; 10 first-frame reads after the click). - Reads that still follow a click on a preloaded task: full detail (`GET /issues/:id`), `work-products`, `documents`, `feedback-votes`, `tree-control/state`, `tree-holds`, `runner-goal`, `browsers`, and the `read` mark. The `chat-bootstrap` payload does not cover them. None gates the chat's first frame (the frame shows at 135 ms while every API response waits 150 ms). - Unit tests: `RecentTaskNavItem.test.tsx` (intent levels, href), `useRecentTaskOpenWarmup.test.tsx` (one request for the top ten, code warm-up, wait for page reads, hidden tab, fresh entries skipped, once per account, user isolation, abort on unmount), `issueChatBootstrapCache.test.ts` (each key in the endpoint shape, `null` versus missing fields, company isolation, no downgrade of newer, in-flight or invalidated data, merge keys filled only when empty). - `ui` Vitest: `src/components src/hooks src/lib src/api` 694 files passed; `src/pages src/App src/context` 188 files passed. `pnpm check:token-gates`: clean. `pnpm verify` exits 2 because a `tests/` file is a global CI input; required CI supplies the broad coverage. ## Risks - Low to medium. One extra read per page load when Recents has tasks without a fresh cached first frame. The read starts only after the page's own queries settle, and the server reads at most two tasks at a time. - A live event that arrives while the preload is in flight, for a task with no cached keys, can leave seeded data up to 30 s old until the next live event or the next open after the stale time. Inbox hover prefetch has the same window. - Classic Recents links now use the identifier route instead of the UUID route. The task page already redirected UUID links to this route. - `tests/perf/collections` is a manual suite. CI does not run it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser automation through Playwright). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 08:00:13 -06:00
DottaandClaude Opus 5.5	15a734d068	fix(heartbeat): show a storage outage wait as live status, then record it once (#20460 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode a run reads and writes file storage on its critical path: it starts its run log (`s3_segments`), restores its saved workspace, and saves it again > - Since #20299 a run waits out a storage outage there for up to 5 minutes instead of failing > - That wait was silent. The board showed "running" with no sign of life, so a storage wait looked like a dead run (restart-chaos R5 `silent_dead_run`) > - The first version of this PR wrote a run event on every wait. That writes the run row (`next_event_seq`, `updated_at`) while the run-log claim is still in progress, and it covered only the run-log begin > - This PR shows the wait as live run status instead, writes nothing during the wait, and records one lifecycle event when the wait ends. It covers every critical-path wait in the run's storage scope > - The benefit is that a storage outage shows on the board as waiting, the run's history keeps one honest line about it, and the claim path is left alone ## Linked Issues or Issue Description Refs #20299 (wait out a store outage on a run's critical path). What happened? - `w-20261110T043533-s11936` (Daytona R1, ST1 194 s, K2): run `d49f58d8` waited 186 s in `heartbeat.prepare.run_output` with no event. The grader failed it as `R5:silent_dead_run`. - `w-20261110T105903-s11936`: the restore download (`storage.snapshot.download`) waited 182 s with no sign of life. Expected: the board shows the run waiting for file storage, and nothing about the notice can disturb run-log ownership. On the failed `run_output` seen on builds of the first version (`w-20261110T101517-s11936` t01 run `9d39550a`, `w-20261110T081647-s11936` t02 run `9da10424`): the evidence points to an expired restart lease, not to the notice's row writes. Neither the writer-authority check (`run-log-authority.ts`) nor the journal claim compares `next_event_seq` or `updated_at`. A remote `bootstrap_incomplete` restart claim gets a 60 s lease (`REMOTE_CONTROLLER_LEASE_MS`), and preparation renews it only for `reattach_remote_runner` claims. In t01 the claim was at 10:19:55, the outage ended at 10:22:18.15, and `run_output` failed at 10:22:18.17, on the first authority check after the store came back. That fix ships separately as its own PR. ## What Changed - `storage/storage-unavailable.ts`: - The run's storage scope can carry a wait observer (`reportStorageOutageWaitsTo`). - `retryThroughStorageOutage` tells it about each backoff (`waiting`). The operation that owns the wait reports one `ended` (`recovered` true or false), so nested waits do not double-report. - `storageOutageWaitFor(purpose, work)` names the step. - An observer that throws never changes the wait. `endStorageOutageWaitOn` keeps the observer. - `heartbeat/storage-wait-notice.ts` (new): - Each wait sets live runtime status (phase `waiting_for_storage`, "Waiting for file storage before starting the run log (45s so far)") and publishes `heartbeat.run.progress`. Nothing goes to the database. - When the wait ends, it writes one `lifecycle` warn event: "Waited Ns for file storage before …", with payload `reason: "storage_outage_wait"`, `purpose`, `waitedMs` and `recovered`. - `heartbeat.ts`: installs the notice next to `endStorageOutageWaitOn` (one line). - Purposes: - run-log begin: "starting the run log"; - `loadWorkspaceSnapshotArchive`: "restoring the workspace"; - `uploadWorkspaceSnapshot`: "saving the workspace". - `packages/shared`: adds `waiting_for_storage` to `HeartbeatRunStatusPhase`. - `tests/restart-chaos/live-status-evidence.ts` (new): R5 also counts the board's live run status (`/api/issues/:id/live-runs` `currentStatusUpdatedAt` / `lastEventAt`) as evidence that a running run is alive. Wiring it into `observer.ts` takes one import and one changed line. - `doc/run-log-events.md`: documents the live status and the one event. - Removed: the first version's per-wait run event. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-output.test.ts src/__tests__/storage-outage-wait.test.ts src/services/remote-workspace-snapshot-storage.test.ts src/services/object-byte-journal.test.ts src/__tests__/storage-s3-credential-rotation.test.ts`: 103 pass. - Red/green: with the observer disabled (master behaviour), these new tests fail. With this change they pass: - run-output "shows the wait as live status and writes nothing to the run while its log claim waits": no run-row write and no event during the claim's waits; the live status is `waiting_for_storage`; the claim's writer-authority check still passes; one lifecycle event afterwards; - the helper's observer tests (one end per wait; gave-up reporting; a throwing observer; the Stop signal keeps the observer); - snapshot storage "tells the run it is waiting for storage while a save or a restore waits out an outage". - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/live-status-evidence.test.ts tests/restart-chaos/observer.test.ts tests/restart-chaos/triggers.test.ts`: 65 pass. `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. `cd server && npx tsc --noEmit`: pass. - Live restart-chaos waves: results in a comment below. ## Risks - Low. - The retry policy, its budget and Stop handling are unchanged. - During a wait, the only new effect is an in-memory status and a live event. - Afterwards, one event per wait, written without blocking the run. - Runtime status is per server process. A board on another instance sees the waiting line only after the closing event. - Shared journal work (`runSharedStorageWork`) keeps its own scope without the observer. Its waits surface through the enclosing run-scoped wait, as before. - Previous-release state: this persists no new run or session state. A run saved by the previous release resumes unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 07:42:43 -06:00
DottaandClaude	c874cfa4da	test(restart-chaos): anchor two-server speed samples on the primary's restart (#20535 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed, including how fast runs recover after a server restart > - Two-server faults (MS1–MS3, #20518) record their `readyAt` after the second server stops > - The speed grade used that time as a restart-recovery anchor, also for faults that never restart the primary > - Wave `w-20261110T115048-s302` passed every task but exited 1 on four such samples > - This pull request anchors two-server samples on the primary's own restart > - The benefit is that a two-server wave fails speed only for a real recovery delay ## Linked Issues or Issue Description Refs #20518 (the two-server faults). What happened? In wave `w-20261110T115048-s302` (Codex, B5, MS2/MS3, 4 tasks) every task passed, but the speed grade counted 4 single-run breaches on `restart_recovery_control_plane` (p50 10.9 s, p95 12.8 s, budget p95 4.0 s), one per fault. - MS2 never restarts the primary (it freezes and thaws it), yet it produced a restart-recovery sample. - MS3's sample started at the fault's `readyAt`, after the second server's 33–43 s tail. The primary's replacement had been ready since 1794311949160 (f02) and 1794312728179 (f04), 43.4 s and 33.6 s earlier. - So each sample measured the B5 run's next provider event after the second server stopped, while `bench.sh` ran quietly. Expected behavior MS1 and MS2 produce no restart-recovery sample. MS3's sample starts when the primary's replacement is ready. ## What Changed - `wave-speed.ts`: - New `primaryRestartReadyAt(fault)`: a non-MS fault keeps its `readyAt`; MS1 gives none; MS2 gives none, unless the primary exited after the thaw and was replaced (then the replacement's `healthyAt`); MS3 gives `multiServer.primaryRestart.readyAt`. - `faultRuns` (recovery samples and the retry "restart" phase) uses that anchor and skips faults without one. - `outageWindows` removes only the time the primary could not serve: none for MS1, the freeze for MS2, and for MS3 from the second server's readiness (when the primary is told to stop) to the replacement's readiness. Before, the second server's whole window was removed from run timings. - `collectWaveSpeed` sets a run's restart phase only from faults that restarted the primary. A freeze still subtracts its downtime, but a run it hit before its first activity stays in its cold or warm budget (review finding). - `wave-speed.test.ts`: six tests built from s302's saved receipts, including two `collectWaveSpeed` cases (a run frozen before its first activity; a retry after an MS2 freeze). - README: one note on how the speed grade treats MS faults. Test-only. No longer freeze and no lease change, as agreed. ## Verification - Red against master `wave-speed.ts`: all 6 new tests fail. Green with the change. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 480 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Offline re-grade of s302 with the fix, from its saved `faults.json` and run timings (`t0*.grade.json`): - Restart-recovery samples go from 4 to 2. The MS2 samples (f01, f03) are gone. - The two MS3 samples are anchored at 1794311949160 (f02) and 1794312728179 (f04), 43.4 s and 33.6 s earlier than before. - Outage removed from run timings: 110.0 s and 113.3 s for the MS2 freezes, 5.6 s and 6.5 s for the MS3 restarts. - No s302 run was hit before its first activity (each freeze came about 68 s into its run), so no run changes phase. - The MS3 latency itself needs the run's event list, which a wave does not save for passing tasks. The observer's saved samples show the run's first recorded activity 0.5 s (f02) and 1.6 s (f04) after the replacement was ready. The next provider event comes at or after that, so this is a lower bound, not the graded value. - `restart_recovery_control_plane` has n = 2 < 5, so its row reports "insufficient sample". The wave's speed verdict now depends only on whether each MS3 sample is under 4.0 s, which this offline data cannot confirm. - No new wave (as agreed). ## Risks - Low. Test-harness code only. - After MS1/MS2 a retry of a failed run is graded as an ordinary warm run, since the primary never restarted. - MS1 overlap and the MS2 thaw tail no longer count as outage, so run timings in those windows are graded in full. The primary served during them. - A primary replaced after an MS2 thaw is graded like any restart. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 07:03:15 -06:00
DottaandClaude	be34a2d9d2	test(restart-chaos): two servers on one database (MS1-MS3) and a cross-server invariant (#20518 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) tests how runs survive control-plane faults > - Every wave runs one server, so no wave can reach a race between two servers on one database > - Hosted Cloud deploys by rolling overlap: a new server's startup recovery and sweeps run beside the old server's live runs > - #20508 fixed such a race (another server's orphan sweep failing a run whose owner took its lease back) with a database test only > - This pull request adds three two-server faults (MS1–MS3) and a cross-server invariant > - The benefit is that cross-server ownership races can be reproduced and graded in a live wave ## Linked Issues or Issue Description Refs #20508 (the race this harness can now reach live). What is missing? The harness starts one control plane per wave. A fault replaces or freezes that one server. Nothing starts a second server on the same database while the first still owns runs, so cross-server ownership bugs are found only by database tests. How production runs (checked before choosing the scenarios) - Hosted Cloud runs on Railway with Neon Postgres. A deploy is a rolling A/B overlap: B starts on the same database, A serves until B is ready, then A drains. Defaults are a 5 s overlap and up to 90 s of drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`, lines 374–375). The chaos plan names the same shape as K12 (`doc/plans/2026-11-06-control-plane-restart-chaos.md`, line 103). It also records B's startup recovery hitting a lock on a run A still owned (lines 229–230). - This repository has no replica count or deploy strategy of its own. `docker/ecs-task-definition.json` describes one task, and the real Cloud deploy lives in another repository. The self-hosting docs do not describe several servers on one database. - Server-side ownership: per-boot controller lease of 90 s (`server/src/services/controller-instance-lease.ts`), legacy run lease of 60 s, native executor lease of 20 min, and the orphan sweep in `server/src/services/heartbeat/recovery.ts`. So MS1 models B's overlap beside a live A, MS2 models A paused (for example a host sleep) past its controller lease while B is up, and MS3 models the rolling upgrade itself. ## What Changed - New `tests/restart-chaos/multi-server.ts`: - MS1: B boots on the same database, instance directory, storage and secrets, on its own port. It stays up 60–150 s beside a live A, then gets SIGTERM. - MS2: B boots. A and its tree are frozen (TM2's freeze without aging) for 100–150 s, past the 90 s controller lease, then thawed. B stays up 30 s more, then stops. - MS3: B boots, then A gets K2 and a replacement on A's port. B stops 20–45 s after the replacement is ready. - B is always stopped once started, also when a step fails. - `checkCrossServer`, graded once per wave with an MS fault, over the wave database and the server process windows (pid, start, exit, freezes): - `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while its owner was alive and not frozen (#20508's shape included). The owner's own lease loss counts only without a prior freeze past the lease. - `MS_double_executor`: a run recorded a second runner identity before the first was shown stopped. - Unattributable ownership failures are listed, not charged. - `supervisor.ts`: records every server's process window and freezes, spawns and stops B (`<wave>/server-b.log`), and runs the MS steps. TM freezes are recorded too. - `types.ts`, `scheduler.ts`, `journey-matrix.ts`, `config.ts`: MS1–MS3 fault kinds, seeded holds, and validation (local only; no storm, crash point or K14). - `main.ts`: writes `<wave>/cross-server.json` and fails the wave on a violation. - README section "Two servers, one database (MS1–MS3)", with the production evidence and the limits. Test-only. No `package.json` or `.github` changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 473 tests pass. - New `multi-server.test.ts` (14 tests): option validation and seeded holds; MS1/MS2/MS3 step order and timing with fake hooks; B stopped when a step fails; the invariant for a live owner, a frozen or exited owner, #20508's thaw-then-write shape, lease loss with and without a freeze past the lease, unattributed owners, and double executors. - New `multi-server-db.test.ts` (embedded Postgres): a run still `running` with `native_execution_ownership_unverified` (#20508's marker) is read at its write time and charged as `MS_foreign_failure`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave `w-20261110T105448-s301` on head `a9ec5a4502` (master product code + this harness), `--adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3`: 3 of 3 pass, all crossed their fault. B booted on the shared database in about 10 s each time. MS2 froze A for 104 s; A stayed up after the thaw. Cross-server invariant: 0 violations, 0 unattributed. ## Risks - Test-harness code only. - B runs on the same host and PID namespace as A, so it can check A's PIDs directly, which a Railway replica cannot. These faults are a lower bound on cross-server exposure. The README says so. - MS2 outlasts the 90 s controller lease, not the 20 min native executor lease. - MS faults refuse TM2/TM3 in the same wave, and fail when B exits before its planned stop. - `MS_foreign_failure` uses the run's `finished_at` (or `updated_at` for a still-running row) and the owner recorded on the run. A run whose owner changed after the failure could be misattributed; the evidence file keeps the owner and windows for review. - `MS_double_executor` reads runner identity events; it cannot see a second executor that never recorded one. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 05:45:08 -06:00
DottaandClaude	16c30ac32b	fix(runner): relaunch a provider whose startup only timed out without a reply (#20521 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. After a restart, a new runner relaunches the provider (for example the Codex app server). A durable startup fence stops a second runner from launching a provider while an unadmitted one may still run. > - #19784 lets a new runner retire a fence whose recorded provider process and group have provably exited. That rule covers a launch that a kill cut off. It deliberately keeps a failed initialization closed, because that can be a real provider failure. > - A relaunch whose reply never came within the runner's 30 s request deadline is saved as the same failed initialization. A stalled or frozen host causes this. The restart-chaos fault TM2 SIGSTOPs the runner and Codex, and their monotonic clocks keep running while they are stopped. > - Every later runner then refused to restore with "provider startup ownership remains unadmitted", and the user had to click Retry. The owner's rule is that a run interrupted by a restart or sleep must resume by itself. > - This pull request records whether initialization failed only because the provider gave no reply, and lets a later runner retire that fence once the provider is provably gone. > - The benefit is that a stall during a provider relaunch continues automatically, while a real provider rejection still fails as before. ## Linked Issues or Issue Description Refs #19784 What happened? Restart-chaos wave `w-20261110T091152-s426`, task t03 (Codex, B5), run `c0bbce18-836b-43af-8195-92ebe3e21cca`. A TM3 restart relaunched the Codex provider. A TM2 freeze stopped the tree 1 s later for 45 s, and a TM3 kill followed the thaw. The next three runners each failed `runner.drain` with "provider startup ownership remains unadmitted". Recovery stopped with `native_session_retry_exhausted`, and the task needed Retry. What did you expect? The new runner relaunches the provider and the run continues. Steps to reproduce The new test `restart_relaunches_a_provider_whose_relaunch_timed_out_without_a_reply` (runner-core `tests/codex_provider.rs`). The fake Codex stalls its first `thread/resume` reply past the 30 s deadline. On `origin/master`, the saved attempt is `initialization_failed` / `thread_open`, and the next runner fails with "provider startup ownership remains unadmitted". ## What Changed Design: - The deadline that passes is the runner's own 30 s request deadline. When a startup request (`initialize`, `thread/start`/`thread/resume`, `thread/read`) reaches that deadline with the provider still running and silent, the runner sets `noReply: true` on the durable `initialization_failed` startup record. - Not `noReply`: a provider error response, a provider that exited before it replied, or closed stdout. - `retire_orphaned_startup_attempt` retires an `initialization_failed` fence only when `noReply` is set and the recorded process and its process group are both gone (`ESRCH`). This is the same proof as `spawned` in #19784. A rejection, an exit, a live process or group, and spawn or admission failures stay fenced. - Validation accepts `noReply` only on `initialization_failed` at the `initialize`, `thread_open` or `thread_read` stage. - Bounded: a runner restores once (`restore_checked` / `restore_error`), so the loop is one relaunch per recovery attempt. Recovery's existing resume budget (`native_session_retry_exhausted`, seen after three attempts in the wave) stops a provider that never replies. - Upgrade: `noReply` is `#[serde(default, skip_serializing_if = "Not::not")]`. A new runner reads state saved by master as `noReply: false`, so that state keeps today's behaviour and stays fenced. - Rollback: an old runner (`deny_unknown_fields`) rejects state that holds a `noReply` fence and fails closed. That run needs Retry, which is today's outcome for the same fence. The field is written only while a timed-out fence is present, and admission clears it, so all other state is unchanged for old runners. - Normal path: nothing is added before `turn.start`. The cost is one in-memory flag per request; no extra I/O. Files: - `runner-core/src/codex_provider.rs`: `last_request_got_no_reply` and the `no_reply` field on `ProviderStartupObservation::Failed`. - `runner-core/src/provider_backend.rs`: the `noReply` field, its validation, and the retirement rule, with unit tests: - a no-reply fence is retired at `initialize`, `thread_open` and `thread_read`; - it is kept while the provider may still run; - a rejection or a state saved without the field stays fenced. - `runner-core/src/bin/fake-codex-app-server.rs`: the test-only switch `--stall-first-thread-resume-ms`. - `runner-core/tests/codex_provider.rs`: the end-to-end red/green test. - `tests/restart-chaos/README.md` and `time-faults.ts`: correct the TM2 claim. A stopped process's monotonic clock keeps running, so TM2 is also a long process stall. ## Verification - `cargo test -p paperclip-runner-core --test codex_provider restart_relaunches_a_provider_whose_relaunch_timed_out` is red on `origin/master` (the next runner refuses restore) and green here, in about 31 s. - `cargo test -p paperclip-runner-core`: 1080 passed, 0 failed. This includes `failed_autonomous_restore_and_rollover_keep_their_exact_startup_origin`, where a real "no rollout found" rejection still never restarts. - `cargo fmt -p paperclip-runner-core -- --check` is clean. - A restart-chaos proof wave on seed 426 will follow in a comment. ## Risks - A provider that hangs for good is relaunched once per recovery attempt until the existing resume budget stops it. Before, it failed on the first attempt. This costs at most a few extra 30 s attempts, and they are bounded. - Rollback to an older runner while a `noReply` fence exists fails closed for that one run, which needs Retry, as it does today. - No database, server or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, cargo and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 05:23:30 -06:00
DottaandClaude	c2423e30e8	test(restart-chaos): provider faults across K2/K4 restarts, per-workload message, per-run speed split (#20515 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos`) proves that runs recover when the control plane restarts > - PR #20494 fixed a saved governed wait whose provider terminal was late or lost across a restart. The harness proved it only with a K1 (kill -9) restart > - The harness could not restart with K2 (graceful SIGTERM) or K4 (outage hold) after a provider fault, so the graceful shutdown drain was not tested > - S1 needs the `interrupt_ack` message and Q1 needs `terminal`, but one wave could set only one message > - G1 was listed as a provider-fault workload, but its agent ends its own turn. No rule ever applies, and each G1 task waits until its deadline > - Speed breaches could not be explained after a wave, because the instance database is deleted > - This pull request adds K2/K4 provider-fault restarts, a message for each workload, removes G1, and keeps a speed split for each run > - The benefit is that one wave can test #20494 across graceful and held restarts. Also, a cold-start breach now shows whether Paperclip or the provider was slow ## Linked Issues or Issue Description Refs #20494 What happened? The harness accepted only `--provider-fault-restart K1`. A wave with `--workloads G1` and a provider fault planned tasks that could not apply the fault. Waves w-20261110T050407-s206, w-20261110T063253-s208 and w-20261110T085554-s211 show `applied=0` for G1 with Codex and Claude. The board then waited for a restart that did not come. What did you expect? A provider fault can cross every restart kind that the harness supports. The harness rejects a workload that does not interrupt a provider turn. Steps to reproduce Run `--adapters codex --workloads G1 --faults PV1 --provider-fault-restart K1 --concurrency 1`. The Codex shim logs `bound` and then `exited` with code 0. It logs no `interrupt_requested`. The task fails `R1:no_rest_within_budget`. ## What Changed - `--provider-fault-restart` accepts K1, K2 and K4. A comma list (for example `K4,K4,K2,K2`) cycles across PV1/PV2 tasks in plan order. K4 holds the control plane down for 120 s. - `--provider-fault-message` accepts `WORKLOAD=message` entries next to the default, for example `S1=interrupt_ack,terminal`. - G1 is removed from `PROVIDER_FAULT_WORKLOADS`. The config check that told G1 to add a restart is removed, because G1 is now rejected. - `speed.json` keeps a `runs` list. Each entry has `bootMs` (Paperclip share), `queueMs`, `providerFirstActivityMs` (turn accepted to first activity) and the pre-turn spans. - README documents the new flags, the G1 limit, and the speed split. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 434 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave w-20261110T092735-s213 (Codex, seed 213, concurrency 1). All 4 tasks pass R1–R6, T and `provider_fault_applied`: - t01 S1 PV2 drop `interrupt_ack` + K4 (120 s hold) - t02 Q1 PV1 delay `terminal` 18 s + K4 (120 s hold) - t03 S1 PV1 delay `interrupt_ack` 18 s + K2 - t04 Q1 PV2 drop `terminal` + K2 - That wave wrote the new `runs` split in `speed.json`. Cold boot was 1.3–1.7 s and provider first activity was 2.0–2.6 s. ## Risks - Low risk. Only the test harness changes. Product code does not change. - Old `plan.json` files keep `providerFaultRestart: "K1"`. That value is still valid, so replays of old waves do not change. - A wave that used G1 with provider faults now fails at parse time. Before, it ran and failed at the deadline. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 05:14:02 -06:00
DottaandClaude	7964433cd0	test(restart-chaos): arm a provider-fault Stop on the first live provider turn (#20512 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) injects provider message faults (PV1–PV3) to test recovery > - A board Stop task (S1) with a PV `interrupt_ack` fault needs the Stop to reach a live provider turn > - The harness clicked Stop only after a seeded wall-clock window, so a fast agent finished first and nothing was injected > - That task then failed `provider_fault_applied`, which made a clean wave exit 1 with no product failure > - This pull request arms the Stop on the first live provider turn for PV tasks > - The benefit is that PV S1 tasks inject their fault every time, so a wave result means what it says ## Linked Issues or Issue Description Refs #20498 (the product fix whose live waves showed this harness gap). What happened? In waves `w-20261110T082702-s212` and `w-20261110T091057-s213` (Claude, `--workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack`), task t03 (S1) failed only `provider_fault_applied` with `applied=0`. Its run finished before the seeded Stop window, so the Stop was missed and the fault never fired. Every product rule (R1–R6, U1) passed. The wave still exited 1. Expected behavior A PV S1 task injects its fault on every run. A missed injection is a harness gap, not a product failure. Steps to reproduce Run `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4`. The second S1 task often finishes before its Stop window. ## What Changed - `observer.ts`: the run-event cursor now tracks a live provider turn (`turn.started` with no `turn.completed/failed/cancelled/interrupted` yet). Run samples expose it as `providerTurnActive`. - New `stopArmed()`: a task with a provider fault arms its Stop on the first poll that shows a live provider turn. Other tasks keep the seeded wall-clock window, unchanged. - A runner replacement or stop clears the live-turn mark, as it already clears open tools. - A PV Stop still counts as missed when the run ends with no live turn seen and the task rests after the window. - The continue comment still waits for the seeded window and `STOP_CONTINUE_AFTER_MS`. - README note under the provider fault section. No product code changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. - New tests in `observer.test.ts`: - `S1 with a provider fault: stops on the first live provider turn, not the wall-clock window` - `S1 with a provider fault: does not stop after a turn terminal, and stops once when the next turn starts` - `S1 with a provider fault: records a missed Stop when the run ends before any live turn and the task rests after the window` - `noteLivenessEvent: a runner replacement or stop clears the live provider turn` - `stopArmed: a provider-fault Stop needs a live provider turn; others keep the window` - Red against master `observer.ts`: the first three new tests fail. Green with the change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - No live wave was run (the wave queue was full); the coordinator asked for a test-only change. ## Risks - Low. This is test-harness code only. - A PV S1 Stop can now land earlier in the turn than before. That is the intended test point for `interrupt_ack`. - A turn shorter than the poll interval (`--poll-ms`, 2–5 s) can still be missed. The README states this limit. - If a provider never records `turn.started`, the PV Stop waits until the task rests and is then recorded as missed, the same result as before. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 04:09:54 -06:00
DottaandClaude	1a89fca1ce	test(restart-chaos): add --rr-stop-on recovering so RR follow-up stops land inside recovery (#20511 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - #20480 added repeated restarts (RR1–RR3): a K1, then more restarts while the replacement recovers. > - In the first four live waves, no follow-up stop landed inside recovery. Every stop saw `/api/ready` report `ready`. > - The saved receipts show why. `/api/ready` reports `recovering` only 15–80 ms before `/api/health` answers, and `ready` 0.25–0.5 s after it. A gap measured from health, even 1 s, always lands after recovery. > - This pull request adds `--rr-stop-on recovering`. Each follow-up stop fires on the first `recovering` status, polled from the replacement's spawn. > - The benefit is that RR waves now test a restart that interrupts recovery itself. ## Linked Issues or Issue Description Refs #20480 What is the problem? RR follow-up stops wait a gap from the replacement's health. On a local host recovery ends about 0.3 s after health, so no stop in four waves (14 tasks, 18 follow-up stops) landed inside recovery. What is the solution? Add `--rr-stop-on recovering`. It replaces the gap: the harness polls `/api/ready` every 50 ms from spawn and stops the replacement on its first `recovering` status. This is the phase poll that K6 already uses. Why is it needed? The deploy loops of 2026-11-08 and 2026-11-09 restarted the server while it recovered interrupted runs. The harness must reach that window to test it. ## What Changed - `repeated-restarts.ts`: `--rr-stop-on recovering` is parsed by `parseRepeatedRestartOptions`. It needs an RR fault and does not combine with `--rr-gap-seconds` or `--replay`. `runRepeatedRestarts` takes `stopOn`: with it, no gap is slept, each stop waits on the new `waitForStopPhase` hook, and only the last replacement is waited on to health. - `supervisor.ts`: `waitForStopPhase` reuses the K6 phase poll (`waitForCrashLoopPhase`, 50 ms). The record carries `stopOn`. - `types.ts`, `config.ts`, `main.ts`: the `rrStopOn` option, its flag, and one line to pass it to the supervisor. - The evidence shows each stop's `/api/ready` status (`phaseAtStop`). With `stopOn`, `gapMs` is null, and the earlier entries have no `healthyAt`. - README: the option and the measured timings. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. New tests cover option parsing, stops on the first `recovering` with no sleep, health waits only on the last replacement, a `ready` stop when recovery was too short to see, an exit before a status, and a missing watcher. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - A live wave from this branch (`--workloads A1,B5,C1 --faults RR1,RR3 --rr-stop-on recovering`, codex, 4 tasks) follows; its results go to a comment. ## Risks - Harness-only change; no product code changes. RR waves without the flag are unchanged. - A stop on `recovering` lands before the replacement answers health. This is the K6 behavior, and the server log shows it. A replacement that never reports a status fails the fault after the 5-minute ready timeout, as K6 does. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 03:56:25 -06:00
DottaandClaude Opus 5.5	7eae4bc2ea	test(restart-chaos): add ST4 partial multipart and ST7 read-after-write lag storage faults (#20504 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` puts a fault proxy between the server and a loopback MinIO (ST1–ST3, ST5/ST5R) > - Remote workspace snapshots over 16 MiB upload as S3 multipart uploads, and saves are read back right after they are written > - No fault hit single upload parts or the completion, and no fault made a just-written object briefly invisible > - Nothing checked that a wave left no orphaned multipart uploads or that every committed snapshot reads back > - This pull request adds ST4 (partial multipart) and ST7 (read-after-write lag), and a teardown store audit > - The benefit is that waves can prove snapshot saves and restores survive part failures and read lag without data loss or leaked uploads ## Linked Issues or Issue Description What existing behavior does this improve? The restart-chaos storage faults treat every request the same. They cannot target one multipart part or the completion, and they cannot lag reads behind writes. Subsystem affected tests/restart-chaos (storage chaos). Current behavior A wave cannot fail the Nth `UploadPart` of a snapshot upload, cut a part body mid-transfer, or fail `CompleteMultipartUpload`. A wave cannot make a `GET` or `HEAD` answer `NoSuchKey` right after a `PUT`. No wave checks for incomplete multipart uploads or unreadable committed snapshots. Proposed behavior `--storage-faults ST4` fails a seeded subset of snapshot upload parts (500/503, reset, or reset after 1–4 MiB of the body), and sometimes the completion. `--storage-faults ST7` answers 404 `NoSuchKey` to reads of an object for 5–30 s after its write succeeded. With either fault, the wave reads every committed workspace snapshot back from MinIO (size and SHA-256 against its row) and, after the server stops, lists incomplete multipart uploads under the wave prefix. Either problem fails the wave. Reason and benefit These are the storage failures that can lose a workspace or leak storage cost without any visible error. ## What Changed - `storage-s3-ops.ts` (new): names the S3 operation from the method and query, and reads the part number. - `storage-multipart.ts` (new): ST4 rule planning. - `storage-lag.ts` (new): ST7 rule planning, and a tracker of successful `PutObject` and `CompleteMultipartUpload` writes per key. - `storage-audit.ts` (new): teardown audit (committed snapshot read-back, incomplete multipart uploads), `storage-audit.json`, and a `Storage audit` section in `SUMMARY.md`. MinIO lists multipart uploads only for an exact key or no prefix, so the audit lists the wave-owned bucket and filters by the wave prefix. - `storage-proxy.ts`: rules can match `operations` and `partNumbers`. New faults `truncate` (reset after N body bytes) and `read_after_write_lag`. `storage-proxy.json` reports `lagged`. - `storage-chaos.ts`, `main.ts`: registry lines for ST4/ST7 and the audit calls. - README: ST4/ST7 rows and the audit. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 423 tests pass. The new `storage-multipart-lag.test.ts` fails before this change (the modules and kinds do not exist) and passes after. It includes real MinIO tests with the AWS SDK: part 2 fails once with 500, a part body is cut after 1 MiB, the completion fails once, the audit finds the incomplete upload and then finds none after completion; a just-written object answers `NoSuchKey` to `GET` and 404 to `HEAD` until the lag passes; the snapshot check reports `sha256_mismatch` and `object_missing`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves run from master after merge (local ST7, Daytona R1 ST4 and ST7). Results go in a follow-up comment. ## Risks - Test harness only. No product code changes. - The audit runs only on waves with ST4 or ST7, so other storage waves do not change. - The snapshot check verifies the stored ciphertext against its row. It does not decrypt, because the key lives in the instance. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 03:49:17 -06:00
DottaandClaude	82abb3ea04	fix(native): classify a Codex API-key unknown model as a model rejection (#20488 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner agents run on a provider (here Codex) with a model the user saves on the agent > - When the provider rejects that model, Paperclip should classify the run as `native_provider_model_rejected`, ask the user to choose a model, and let Retry finish the task > - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON envelope). With API-key auth, Codex returns a different terminal error for an unknown model > - So the run failed as a generic `adapter_failed` ("Native session failed"), and recovery opened a reconciliation incident ("Recovery needed") for a turn that never did any work > - This pull request recognizes the real API-key shape, bound to the configured model, and names the rejected model on the task > - The benefit is that the task asks for a model choice instead of a reconciliation, and Retry completes the task after the user picks a valid model ## Linked Issues or Issue Description What happened? The restart-chaos config fault CF3 `model_id` saves a Codex agent with the model `chaos-missing-model-<nonce>`. The run stays "running" for about 60 seconds, then fails with `errorCode: adapter_failed`, "Native session failed". The task shows "Recovery needed", cause `native_continuation_requires_reconciliation`. The grader fails as `CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01 and `w-20261110T053831-s205-cf` t02). Root cause: the wave runs Codex with an OpenAI API key. I drove the bundled `codex app-server` (0.160.0) directly with an unknown model: - With API-key auth, Codex retries the Responses API 404 ten times (about 59 s), then fails the turn with `{"message":"unexpected status 404 Not Found: The model \`<id>\` does not exist or you do not have access to it., url: https://api.openai.com/v1/responses, cf-ray: …, request id: req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`. - With ChatGPT auth, the turn fails in about 3 s with `message` = `{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The '<id>' model is not supported when using Codex with a ChatGPT account.","code":"model_not_found"}}`. Paperclip already classified this one. `createNativeProviderFailureObservation` parsed only the JSON envelope, and the runner's provider-failure category for an HTTP 404 is `unknown`. The terminal therefore stayed unclassified. Expected behavior The run is `native_provider_model_rejected`, the task shows `userAction: "choose_model"` and names the model, no reconciliation incident opens, and Retry completes the task after the user picks a valid model. Steps to reproduce `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex --references model_id --tasks 1 --concurrency 1 --restarts K2 --seed 206` on `master` (with `OPENAI_API_KEY` set). Paperclip version or commit `master` at `5ab98b0`. ## What Changed - `server/src/services/native-runtime/native-provider-failure.ts`: the Codex observation recognizes the API-key terminal shape (`codexErrorInfo` `httpConnectionFailed` or `responseStreamDisconnected` with `httpStatusCode: 404`, and the message prefix for the configured model only). It classifies it as `native_provider_model_rejected` with a new closed diagnostic `NATIVE_MODEL_NOT_FOUND_DIAGNOSTIC` (`codex`, `model_not_found`, `404`, `api_key`). No provider text or model name is stored in the diagnostic. A recoverable turn, another model's 404, or another status is not classified. - `readNativeModelRejectionDiagnostic` accepts both closed diagnostics, so run failure diagnostics show it. - `TaskInterruptionPresentation.rejectedModel` (shared type): with `userAction: "choose_model"`, the task query carries the model id from the run's native execution input. Only a `native_provider_model_rejected` run discloses it. - `TaskChatThread` names the model in the "Model unavailable" marker: `The provider rejected the model "<id>".` - The restart-chaos config wave reads `interruption.rejectedModel` as task-level text. For `choose_model`, its repair step uses the Retry that the "Model unavailable" marker offers (`POST /agents/{id}/wakeup` with `reason: "retry_failed_run"`, the same request as `agentsApi.retryFailedRun`), because a model rejection does not offer the task-level Retry. ## Verification - Red before, green after (the classification line disabled makes both fail): - `native-provider-failure.test.ts` — "classifies the API-key 404 for the configured model as a model rejection (codex 0.160.0 shape)", with the exact terminal error captured from `codex app-server` 0.160.0, plus negative cases. - `native-provider-failure.integration.test.ts` — new `api-key` scenario through `executePaperclipNativeSession`, the persisted evidence reader and `finalizeNativeRun`: the run ends `native_provider_model_rejected`, the issue is `blocked` with that decision, and no issue recovery action is created. - `task-interruption-query.test.ts` — the real task query carries `rejectedModel` only for a model rejection. `TaskChatThread.test.tsx` — the marker names the model. - `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck` pass. - Live proof: config wave `w-20261110T090801-s206-cf` (CF3 `model_id`, codex with an OpenAI API key, K2, seed 206) on commit `05e8938`: 1/1 pass. The run ends `native_provider_model_rejected`; the task rests as `waiting_for_user` / `choose_model` with `rejectedModel: "chaos-missing-model-472e1851"` and execution cause `native_provider_model_rejected` (no reconciliation incident), rests again after the K2 restart, and finishes 27 s after the model is restored and the marker's Retry is used. On `master` the same wave failed as `actionable_message` with "Recovery needed" / `native_continuation_requires_reconciliation` (`w-20261110T061914-s206-cf`). ## Risks - Low risk. The match is bound to the configured model, a 404 status and a non-recoverable turn, so other provider failures keep their current behavior. - The match depends on the Codex terminal text. If a later Codex release changes it, the run falls back to the old generic failure, not to a wrong classification. - Claude (ACPX) unknown models are not changed here. Claude Code reports an unknown model as a synthetic assistant message ("There's an issue with the selected model …") rather than a provider error; this PR does not change or live-verify that path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, a direct `codex app-server` probe). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 03:11:10 -06:00
DottaandClaude	182a7cd4fe	test(restart-chaos): keep process start identity out of TM2/TM3 aging (#20500 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) checks recovery under faults. TM2 (host sleep) and TM3 (aged outage) move every database time back by an age. > - The aging also moved process start times: `controller_process_started_at`, `process_started_at`, and `processStartedAt` copies in JSON. > - A process start is identity, not a deadline. The server reads its own start from the OS once and compares it with the stamp an owner saved. A real sleep never changes it. > - After a shift, every live owner looked like a different process and failed its identity check, whatever its lease said. This harness artifact hid whether a lease fix works (#20474). > - This pull request keeps process start identity unaged and makes the grader's unshift skip it too. > - The benefit is that TM2/TM3 failures now come only from deadlines that really lapse, as on a real laptop. ## Linked Issues or Issue Description Refs #20458 (adds TM1–TM3) and #20474 (the lease fix whose proof wave exposed this). What happened? A TM2 wave against #20474 still failed at the 300 s renewal. The fix takes back a lapsed lease only for the same owner (token, attempt, generation, boot, pid and process start). The aging had moved the saved `controller_process_started_at` back by one hour. The live process still reported its real start, so the identity comparison failed. Expected behavior Aging moves deadlines and event times. It never moves facts that a real sleep cannot change, such as when a process started. Steps to reproduce Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults TM2 --time-age-hours 1 --concurrency 1 --tasks 2 --seed 304 --server-from <#20474 checkout>`. Before this change it fails 0/2 (w-20261110T072106-s304). ## What Changed - `time-faults.ts`: add `isProcessIdentityTime`. `agingStatements` skips columns named for a process start. `paperclip_shift_json` keeps JSON keys named for a process start (any spelling) unchanged. `unshiftJson` skips the same keys, so the grader does not move them forward. - `time-faults.test.ts`: a unit test for the statements and the unshift. - `time-faults-db.test.ts`: the embedded-Postgres aging test now checks that a `controller_process_started_at` column and `processStartedAt` / `processStartedAtMs` JSON keys keep their values. - `README.md`: one paragraph under Aging that says why. ## Verification - `npx vitest run -c tests/restart-chaos/vitest.config.ts tests/restart-chaos/time-faults-db.test.ts tests/restart-chaos/time-faults.test.ts`: 26/26 pass. Without the `time-faults.ts` change, 2 fail (the new unit test, and the DB aging test's identity check). - `npx tsc --noEmit -p tests/restart-chaos` is clean. - Test-only change. No product code. ## Risks - Low. Only the harness changes. A column or key whose name contains "process started at" is no longer aged. If such a field were really a deadline, it would no longer lapse, but none is. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 02:43:55 -06:00
DottaandClaude Opus 5.5	2f59b9c67c	test(chaos): fire a planned restart while the server is mid S3 transfer (#20451 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness restarts the control plane while agent runs are in flight, and `--storage s3` runs the server against a loopback MinIO behind a fault proxy > - Planned restarts fire by time or by run events. A time-based restart seldom lands while the server is moving bytes to or from the store > - The riskiest storage moments are a workspace snapshot upload, a snapshot restore, and a run-log journal write. A restart in the middle of one must not lose or corrupt the workspace or the log > - This pull request adds a storage-triggered restart: the planned fault fires the moment the proxy sees a matching S3 request, and the proxy holds that request so the restart interrupts it > - The benefit is that restart-during-transfer becomes a repeatable wave with receipts that name the interrupted request ## Linked Issues or Issue Description Subsystem affected Restart-chaos test harness (`tests/restart-chaos`), cloud storage mode. Problem or motivation A K1/K2/K4 restart fires on a timer or on a run trigger. With `--storage s3`, the timer almost never lands during a snapshot upload, a snapshot download, or a run-log write. So the harness does not prove that a restart in the middle of a transfer is safe. Proposed solution Add `--storage-trigger <target>`. At its planned time, each task's fault is armed. The next server request that matches the target fires it. The proxy holds the request open (up to 60 s) while the harness snapshots the tasks and restarts the server. Alternatives considered A new `--triggers storage` id. It needs edits to the observer (near its 1,000-line limit), the trigger registry and the delay table. The storage trigger is a wave-level mode in a new module instead. ## What Changed - New `tests/restart-chaos/storage-trigger.ts`: targets `snapshot-upload`, `snapshot-part` (multipart `UploadPart`), `snapshot-download` and `run-log`; the `StorageTrigger` that arms faults, fires on a match, holds the request and records receipts. - `storage-proxy.ts`: an `observe(listener)` hook. The listener sees every request and can hold it (the existing latency path). - `main.ts`, `config.ts`, `types.ts`: registry lines only. `--storage-trigger` (needs `--storage s3` and `--triggers time`); a due fault is armed instead of fired; the wave writes `storage-trigger.json`. - `storage-trigger.json` records the matching request (method, path with upload ids hidden, declared bytes, outcome `aborted` or `completed`, status), every other request in flight, and the fault id. A fault that never matched is recorded with `missedReason`. - README: "Restart during a transfer". ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (25 files, 304 tests pass). - Red/green: `storage-trigger.test.ts` against the master `storage-proxy.ts` fails 3 of 6 tests (no hook, so nothing is held or fired); it passes with this change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local run-log, Daytona snapshot upload and download) are recorded in a PR comment as they finish. ## Risks - Low risk. Test harness only. Without `--storage-trigger`, the proxy has no listener and behaves as before. - A replay re-arms its faults at their observed offsets. It fires on the next matching request, not on the exact same request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:40:54 -06:00
DottaandClaude Opus 5.5	7fef6321fe	test(restart-chaos): remove storage fault window time from speed budgets (#20501 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed against local budgets (`wave-speed.ts`) > - The speed grade already removes restart outage time (fault start to server ready) from spans and restart recoveries > - It did not remove storage fault window time (ST1 outage, ST5 rejected credentials, and so on) > - A restart recovery that correctly waits for the store inside a storage fault window then counts as a speed breach > - This pull request removes the overlap with storage fault windows the same way, and shows the removed time on its own line > - The benefit is that storage waves fail only on real slowness, not on correct waiting ## Linked Issues or Issue Description What existing behavior does this improve? The wave speed grade treats time spent inside a storage fault window as product slowness. Subsystem affected tests/restart-chaos (speed grade). Current behavior The ST5 waves w-20261110T050044-s503 and w-20261110T062313-s506 exited 1 on speed breaches only. Their `restart_recovery_control_plane` p95 was 62–67 s against a 4 s budget. The recoveries were inside an ST5R window and correctly waited for a valid key. Proposed behavior Spans and restart recoveries do not count time inside storage fault windows (from the `storage-faults.json` receipts). Restart outage and storage windows that overlap are removed once. `SUMMARY.md` shows the storage time that was removed, and `speed.json` has `storageFaultWindows` and `storageFaultRemovedMs`. Reason and benefit Storage waves must fail on real product failures and real slowness, not on the harness's own fault windows. ## What Changed - `wave-speed.ts`: `storageFaultWindows()` turns receipts into windows (an open window ends now). `mergeWindows()` joins overlapping windows, so `outageFreeClock()` removes shared time once. `collectWaveSpeed()` takes the storage receipts and removes their time from spans; `gradeWaveSpeed()` removes it from restart recoveries. The storage share is reported apart from the restart outage share. - `main.ts`: one line passes `storageChaos.receipts` to `collectWaveSpeed()`. - README: one paragraph in the Cloud storage section. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/wave-speed.test.ts`: 20 pass. The 3 new tests fail before this change (`storageFaultWindows is not a function`, overlap removed twice, span not trimmed) and pass after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. ## Risks - Test harness only. No product code changes. - ST2 (slow store) windows are also removed from speed budgets. Slowness during a slow-store window is expected, so this is intended. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:39:09 -06:00
DottaandClaude	a14ae9b5c9	fix(workspaces): name a failed workspace setup command on the task page (#20478 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent starts, Paperclip prepares the task workspace and runs the project's configured setup commands (for example the workspace provision command) > - When that command fails, the run fails as `setup_failed`, and the task stops with a Retry > - The task page shows only "Failed" and a generic recovery notice. The real error (the command, its exit status and output) is visible only on the run page > - The user cannot see from the task which setting is broken or what to change > - This pull request records a bounded, named cause for a failed setup command and shows it on the task page > - The benefit is that the task names the broken setting and what to change, and classification, retry and recovery stay the same ## Linked Issues or Issue Description What happened? The restart-chaos config fault wave CF1 sets a project `workspaceStrategy.provisionCommand` to `bash ./scripts/provision-worktree.sh`, a script that does not exist. The run fails as `setup_failed` with `Execution workspace provision command "bash ./scripts/provision-worktree.sh" failed: bash: ./scripts/provision-worktree.sh: No such file or directory`. The task page shows "Failed" and "Paperclip automatically retried continuation … still has no live execution path". The grader fails the task as `CF1/provision_command:message_only_in_run_detail`. Expected behavior The task page names the failed setup step, the configured command, its exit status and a short tail of its output, and says what to change. Retry behavior stays the same. Steps to reproduce `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks 2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`. Paperclip version or commit `master` at `5ab98b0` (wave `w-20261110T043929-s204-cf`, tasks t01 and t03). ## What Changed - `WorkspaceCommandExitError` now carries the setup step (`workspace_provision`, `workspace_seed`, `workspace_runtime_provision`), the configured command and the command output. - New `server/src/services/workspace-setup-command-failure.ts`: it builds a bounded, redacted `setupCommandFailure` record (step, command up to 200 characters, exit code, last 300 characters of output). It also recognizes the older message format, so runs recorded before this change still get a name. - `run-completion.ts` adds `setupCommandFailure` to the setup failure `resultJson`. The error code (`setup_failed`) and all retry and recovery fields stay the same. - `TaskInterruptionPresentation.setupFailure` (shared type) is set for `setup_failed` runs. It goes through the same per-run secret redaction and user-name redaction as `workspaceFailure`. - `ExecutionBlockerNotice` shows "Couldn’t set up the task workspace. <command> failed with exit code N: <output>. Fix the command in the project’s workspace settings, then retry." A scheduled transient retry keeps its countdown and only adds the name. - The restart-chaos config wave reads `interruption.setupFailure` as a task-level text for its actionable-message check. ## Verification - Red before, green after (the tests fail with the source changes reverted): - `server/src/__tests__/workspace-runtime.test.ts` — "names a failed project provision command with its exit status and output (CF1)" runs a real git worktree with a missing provision script (exit 127). - `server/src/services/task-interruption-presentation.test.ts` — "setup command failure presentation (CF1)": names the command, keeps `state`/`canRetry`/`userAction` identical, recognizes the older message, keeps a scheduled retry, bounds output and strips credentials from the command. - `ui/src/components/ExecutionBlockerNotice.test.tsx` and `tests/restart-chaos/config-faults.test.ts` cover the notice text and the harness check. - `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck` pass. - Live proof: config wave `w-20261110T083227-s204-cf` (CF1, codex, 2 tasks, K1 and K2, seed 204) on commit `ae9a4b7`: 2/2 pass. Both tasks show `The project's workspace provision command "bash ./scripts/provision-worktree.sh" failed with exit code 127: bash: ./scripts/provision-worktree.sh: No such file or directory` on the task, rest again after the restart, and finish after the repair and Retry. On `master` the same wave failed both tasks as `message_only_in_run_detail` (`w-20261110T043929-s204-cf`). ## Risks - Low risk. The change adds a presentation field. It does not change the error code, retry, recovery or blocking behavior. - The command and output are user-configured text. They are bounded, passed through `redactSensitiveText`, the run's secret registry and user-name redaction. If redaction fails, the field is omitted, like `workspaceFailure`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 02:35:18 -06:00
DottaandClaude	1e13c58a4e	fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat run settlement records why a run failed, and the board shows that record. > - Runs that fail without a named code settle as `adapter_failed` (or `setup_failed`) with no cause. > - A user, an operator, and the restart chaos harness cannot tell what went wrong from such a run. > - Some run errors also show raw SQL statement text to users. > - This pull request records a bounded `failureCause` code for generic failures, shows it on the board, and keeps SQL text out of run errors. > - The benefit is that a generic failure now names its cause, for example "adapter failed (40001)", without leaking messages, SQL, or secrets. ## Linked Issues or Issue Description Refs #20385 (the nested-cause extraction this change reuses). Related: #20421 (restart chaos universal checks U1-U4). What happened? The restart chaos audit of 79 saved waves found 30 failed runs with `errorCode: "adapter_failed"` and a null or generic error. Every U3 ("diagnosable") warning was `adapter_failed`. Nobody could tell what failed. Some board run errors also showed raw SQL statement text from wrapped database errors (Drizzle `Failed query: ... params: ...`). Expected behavior A run that fails with a generic code names a short, safe cause. The board shows the cause next to the stop reason. Run error text never shows SQL statements or their parameters. Steps to reproduce Make an adapter throw a wrapped database error (an `Error` whose nested `cause` has `code: "40001"` and whose message has a Drizzle `Failed query:` statement). Before this change, the run settles as `adapter_failed` with no cause, and `heartbeat_runs.error` contains the SQL statement and parameters. The new tests in `server/src/services/heartbeat/run-completion.test.ts` reproduce this. ## What Changed - Runner: move the #20385 event-commit cause extraction into `safeErrorCauseCode` (`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and export it. The Runner's own diagnostics use the same function, unchanged in behavior. - Shared: add the `failureCause` contract (`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]` token of at most 80 characters, `normalizeRunFailureCause`, `readRunFailureCause`, and `withoutSqlStatementText`. - Server: add `server/src/services/heartbeat/run-failure-cause.ts`. Settlement records `resultJson.failureCause` only when the code is generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same. - Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`, else `provider_<category>`, else the first safe code in the error and up to three nested causes (`40001`, `ECONNRESET`), else the error class name, else `unknown_error`. - Adapter results without a code (`completeRun`): the error family, `signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`, `adapter_reported_error`, or `unknown_error`. - The failure log lines (`heartbeat execution failed`, `heartbeat execution setup failed`, and a new `heartbeat run failed` warning for generic adapter results) include `failureCause` and `errorCode`. - Server: replace SQL statement text in run error text with "Database query failed (<cause>)" at all three settlement paths, including the cancelled branch. The execution blocker's `runError`, the attention inbox (failed-run excerpt and metadata), and continuation summaries apply the same replacement on read, so older rows also stop showing SQL there. - Server: Retry admission and failed-preparation replay accept only a closed set of `resultJson` keys. `failureCause` is added to each of those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`, `cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is a diagnostic code with no execution or replay authority, so a failed-preparation run stays retryable. - Server: `runsForIssue` returns a validated top-level `failureCause`. It is null for older rows, non-string JSON values, and malformed values. - UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed (40001)". Older runs keep "adapter failed". The Storybook run-ledger fixture has a failed run with a cause. - Restart chaos harness: run samples keep `failureCause`. U3 treats a generic code with a specific `failureCause` as diagnosable. A bare `Error`, `unknown_error`, or the no-code fallback `adapter_reported_error` still counts as vague. U3 stays a warning. - Docs: `doc/run-log-events.md` and the harness README describe the field and the SQL replacement. ## Verification - Red before the fix: the four new `run-completion.test.ts` cases failed against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL text in `error`). - Green: - `server/src/services/heartbeat/run-completion.test.ts`: all tests passed (real embedded PostgreSQL). New cases cover a nested `40001` three levels deep for execution and setup failures, SQL and a secret parameter absent from `error` and `resultJson`, the bound (an 81-character code and a SQL "code" fall back to the class name), `exit_code_137` for a code-less adapter result, a cancelled result with SQL text, no cause for a named code, and the log line. On this loaded host the suite's `afterAll` database teardown hit its 30 s hook limit after every test passed (the known local teardown limit, also seen in #20385). CI runs the suite. - `server/src/services/heartbeat/run-failure-cause.test.ts` (4), `server/src/__tests__/activity-service.test.ts` (recorded, legacy, malformed, overlong, boolean, and number cause projections), `server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL row). - `packages/shared/src/run-failure-cause.test.ts` (6), `ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos `universal.test.ts` and `observer.test.ts`. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`: 207 passed (the stall-diagnostic cases still read `40001` and `ECONNRESET`). - Typecheck: shared, runner, server (all layers), UI, restart chaos harness. `pnpm check:token-gates`: all gates clean. - `pnpm verify` returned exit 2 (global doc input changed); required CI supplies broad coverage. - CI on `6295f8d` found that the new key made failed-preparation runs ineligible for Retry (`chat-channels.integration.test.ts` saved board-message Retry got 409; `workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries). Both reproduced locally and pass after the allowlist fix, with the other 459 Retry cases in `chat-channels.integration.test.ts` and all 25 in `workspace-recovery-preparation-dispatch.test.ts`. - No chaos waves were run. Storybook was not opened in a browser; the fixture typechecks and the label is covered by the UI test. ## Risks - Low risk. `errorCode` values do not change, so retry, recovery, and comparison logic is unaffected. - No schema change and no migration. `failureCause` is a new optional `resultJson` key. Rows saved by the previous release have no key and render as before; a new row read by an old server is ignored. - The SQL replacement only acts on a Drizzle `Failed query:` marker, or on `$n` placeholder text with a statement shape. Ordinary error text does not change. When it acts, it also drops the text after the statement. - Code that checks `resultJson` against a closed key list must allow `failureCause`. I found and updated the four lists on the Retry path; a list I missed would refuse a Retry for a run with a generic failure. - This is the run log / instance database path. It adds no Telemetry event. - The server failure log line still logs the raw error message (unchanged behavior); only the board-facing text is changed. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file editing) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 02:14:53 -06:00
DottaandClaude	5c9641f141	test(restart-chaos): add repeated restarts (RR1-RR3): a restart inside the previous restart's recovery (#20480 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Every existing fault restarts the server once per task. > - Real incidents on 2026-11-08 and 2026-11-09 were deploy loops. The server restarted again 5–60 s after it came back, while recovery of interrupted runs was still in progress. > - No wave puts a second restart inside the first restart's recovery window. > - This pull request adds a fault family, repeated restarts (RR1–RR3). It does not change product code. > - The benefit is that waves now test recovery that is itself interrupted. ## Linked Issues or Issue Description What is the problem? The chaos harness restarts the server only once per fault. Production deploy loops restart it again while it recovers, and no wave tests that. What is the solution? Add three repeated-restart faults. Each one starts as a K1 and then stops the replacement again after a seeded gap from its health. Why is it needed? Recovery of interrupted runs can itself be interrupted. A second restart in that window can leave a lease, a claim or a wake half done. Refs #20458 (the TM fault family this one follows). ## What Changed - Add `RR1`, `RR2` and `RR3` to `--faults` (`repeated-restarts.ts`): - RR1 double crash: K1, then one more `SIGKILL` 3–20 s after the replacement reports healthy. - RR2 crash, then graceful: K1, then one `SIGTERM` (as K2) 3–20 s after health. - RR3 crash loop: 3–4 `SIGKILL`s in all, 10–30 s after each health. The last replacement stays up. - Add `--rr-gap-seconds <n>` or `<min>-<max>` (1–300). It replaces every RR gap. RR faults do not combine with `--storm-minutes`. - The gaps are seeded per task (`restartSteps` in `plan.json`, also in `--matrix` plans). They are drawn only for RR tasks, so the plans of existing seeds do not change. `--replay` restarts again after the recorded gaps. - The supervisor runs the whole sequence as one fault receipt. The grader therefore sees one fault window, from the first kill to the last replacement's ready. Crossing, R1's outage allowance, R5, the T-oracle, task deadlines and the recovery speed budget treat it as one fault. No check is relaxed. - Each stop and replacement is recorded with its signal, gap, PID, the `/api/ready` status just before the stop, and its stopped, restarted and healthy times. The record goes to `<wave>/repeated-restarts/<faultId>.json` (also written when the sequence fails part way), to the receipt (`repeatedRestart` in `faults.json` and in each failure's `startup-phases.json`), and to the timeline (`repeated_restart_stop`, `repeated_restart_healthy`). - Small registry additions: `types.ts`, `config.ts`, `scheduler.ts`, `journey-matrix.ts`, `main.ts`, `evidence.ts`, `supervisor.ts`, and the README section next to TM. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 33 files, 402 tests pass. The new `repeated-restarts.test.ts` covers option parsing, seeded planning for each kind, unchanged plans for other faults, steps in the task schedule, the matrix and replay, the stop-and-replace loop on a fake clock (gaps measured from health, `SIGTERM` for RR2, an exit during a gap, shutdown), and one crossing window for the sequence. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - `pnpm verify` asks for broader coverage (the README is a global CI input), so required CI supplies it. - Live waves run from master after merge, as the merge rule for test-only harness PRs allows. Their results go to a follow-up comment. ## Risks - Harness-only change; no product code changes. Waves without RR faults are unchanged: the plan draws, the fault receipt and the grader paths are the same. - During an RR sequence the server stays "not ready" for the harness, so no other fault fires and the observer does not click Retry until the last replacement is ready. This is the same rule as for K6. - A replacement that exits on its own during a gap fails the fault as a harness error, with the stops so far recorded. The wave then recovers the server as after any failed fault. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 01:44:51 -06:00
DottaandClaude Opus 5.5	179694de7d	test(chaos): provider message faults for Claude (ACPX) and across a K1 restart (#20454 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) proves that agent work survives control-plane restarts and provider faults > - PR #20432 added provider message faults (PV1 delay, PV2 drop, PV3 duplicate), but only for Codex, and only in waves without restarts > - Claude runs through the ACPX sidecar, whose provider pack and sidecar are digest-verified, so the Codex executable shim cannot reach it > - A late or missing terminal can also arrive across a control-plane restart, which no wave could exercise > - This pull request adds a Claude (ACPX) seam with no product change, and a K1 restart that fires after a provider fault applies > - The benefit is that governed waits, Stop and approvals are now tested under provider message faults for both main adapters, also across a kill -9 ## Linked Issues or Issue Description Refs #20432, #20330, #20336 What is the problem? Provider message faults covered only Codex. They could not run with restart faults. PV3, the `interrupt_ack` target, and the S1 and G1 workloads under PV1/PV2 never ran live. What is the expected outcome? The harness can delay, drop or duplicate one targeted provider message for Claude and Codex. A PV1/PV2 fault can straddle a K1 restart, and the grade states what a pass means. Why is this the right layer? This is test-harness work. The product is not changed. The seam replaces only a file the runner already hashes and verifies at launch, inside a per-wave stand-in package. ## What Changed - `acpx-provider-faults.ts`: builds a per-wave stand-in runner package under `<repo>/node_modules/.restart-chaos-<wave>/` (symlinks to the real package, except a generated `dist/cli/acpx-runtime-sidecar.cjs`). It also parses `--provider-fault-restart`. - `provider-shim/acpx-sidecar-hook.mjs`: a server-only resolve hook that loads `runnerd-codex-transport` (source `src/live/.ts` or built `dist/live/.js`) from the stand-in package. Product code then hashes the stand-in sidecar, and the runner verifies and launches it. The module's own imports resolve to the real package. - `provider-shim/acpx-fault-tap.cjs`: the stand-in installs this tap and then runs the real sidecar bundle in the same process, as the runner's verified CommonJS loader does. For Claude, `terminal` is the provider's ACP `session/prompt` response after a `session/cancel`. `interrupt_ack` is the sidecar's `turn.cancel` response, because ACP's cancel has no acknowledgement. The provider process is still spawned and admitted by product code. - `--provider-fault-restart K1 [--provider-fault-restart-after-seconds n]`: each PV1/PV2 task also kills -9 the control plane n seconds (default 3) after its fault applied, through a new internal `provider` trigger. PV3 and planned K cells stay rejected. - `withProviderFaultCheck`: with a restart, the check passes only if the fault applied once, the K1 started after it, and for PV1 no release came first. The ordinary R1–R6 grade and the workload T-oracle still decide whether the task recovered. - `config.ts`: provider faults accept `--adapters codex,claude`. G1 with a provider fault needs `--provider-fault-restart K1`, because its oracle grades the board decision after a restart. - README: the Claude seam, the target table per adapter, what a pass means across a restart, and the Claude PV1 delay window. - Unit tests: the ACP and runner-pipe taps, the generated stand-in with a fixture sidecar, the hook against a fake runner package, planning, config validation, and the restart check. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (all pass after rebase on master) - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - Live waves (local, one at a time through the wave-slot protocol): \| Wave \| Scenario \| Result \| \| --- \| --- \| --- \| \| w-20261110T040202-s202 \| Claude Q1, PV1 terminal 18 s \| Task passed its Q1 oracle; `provider_fault_applied` failed: the runner settled the cancelled turn ~3.6 s after the cancel and closed the sidecar before the release, so 18 s is a drop for Claude (documented) \| \| w-20261110T041224-s203 \| Claude Q1, PV1 terminal 1 s and 3 s \| 2/2 pass (held and released) \| \| w-20261110T041355-s204 \| Claude Q1 and S1, PV2 terminal \| 2/2 pass \| \| w-20261110T043354-s205 \| Codex Q1, PV3 terminal (duplicate) \| 1/1 pass \| \| w-20261110T050407-s206 \| Codex S1 and G1, PV1/PV2 terminal \| 0/4: S1 terminal never sent (Codex exits after Stop, so `terminal` cannot apply; documented, use `interrupt_ack`); G1 needs a restart by design (now rejected without `--provider-fault-restart K1`). No product failure. \| \| queued \| Codex+Claude Q1/S1 PV1/PV2 `interrupt_ack`; Codex+Claude Q1/G1 PV1/PV2 + K1 \| Runs from master after merge (queue is ~90 min deep) \| ## Risks - Low product risk: no product code changes. - The seam depends on the runner computing the local sidecar path from the transport module URL and on the ACPX provider bootstrap marker string. If either changes, the tap logs nothing and every Claude PV task fails `provider_fault_applied`. It does not pass silently. - The stand-in package is written under the workspace `node_modules` (gitignored) and removed at wave end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 01:06:25 -06:00
DottaandClaude	bf1524c5f8	test(restart-chaos): pin each wave's API URL to its own loopback server (#20462 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness starts an isolated Paperclip server for each wave and runs real agents against it > - Agents find the server through `PAPERCLIP_API_URL`, which the server derives at startup > - On a developer machine with a public URL for the main instance (for example a tailnet origin), the wave server derived that public URL instead of its own loopback origin > - So `process` agents sent the run key for the temporary instance to a different host, which returned 502 > - This pull request pins each wave server's `PAPERCLIP_API_URL` to its own loopback origin and removes the Q3 script workaround > - The benefit is that a wave's agent credentials only go to the wave's own server ## Linked Issues or Issue Description What happened? In isolated restart-chaos wave instances, `process` agents received `PAPERCLIP_API_URL` set to the machine's public tailnet origin. That origin serves the developer's main instance, not the wave. The agent's run key for the temporary instance was sent to that host, which returned 502. The Q3 agent script worked around this with a hard-coded loopback base URL. Expected behavior A wave's agents call the wave's own server. Steps to reproduce 1. Configure a public URL (for example a tailnet origin) for the developer's main Paperclip instance on the machine. 2. Run a restart-chaos wave with a `process` agent that reads `PAPERCLIP_API_URL`. 3. The agent's API calls go to the public origin, not to the wave's `127.0.0.1` port. Paperclip version or commit master at `2c2186706e` Additional context This is a harness isolation gap, not a product bug. The server intentionally lets an explicit `PAPERCLIP_API_URL` win, and otherwise prefers the configured public URL (`choosePrimaryRuntimeApiUrl`), because an operator's public URL is the address agents must use in many deployments. The isolated wave server is the component that must name its own origin. `tests/runner-e2e/server.ts` already pins its public URL to loopback in its public-MCP mode for the same reason. ## What Changed - `tests/restart-chaos/supervisor.ts`: new `waveServerUrlEnvironment(port)` sets `PAPERCLIP_API_URL` to `http://127.0.0.1:<port>` in every wave server environment (also after restarts and a K14 switch). - `tests/restart-chaos/journey-driver.ts`: the Q3 agent script reads `PAPERCLIP_API_URL` again. The hard-coded loopback workaround is removed. - `tests/restart-chaos/journeys.test.ts`: two tests. ## Verification - Red before, green after (`npx vitest run -c tests/restart-chaos/vitest.config.ts tests/restart-chaos/journeys.test.ts`, 22 passed): - `pins the wave's loopback origin over an ambient public API URL` - `sends the Q3 agent's run key only to the PAPERCLIP_API_URL its server assigned` (runs the real Q3 script against a local HTTP server and checks that every request goes there with the run key) - `npx tsc --noEmit -p tests/restart-chaos/tsconfig.json` passes. ## Risks - Low risk. Test harness only. No product code changes. - The pin applies to `PAPERCLIP_API_URL` only. A public URL from the developer environment can still set the wave's auth base URL. No current workload depends on it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 00:24:21 -06:00
DottaandClaude	400bb1f0f9	test(restart-chaos): one-row db pressure, bound accepted-Retry waits in U1, live L1 results (#20472 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness grades real agent tasks on an isolated control plane. > - #20421 added the L1 contention workload, `--db-pressure`, and the universal checks U1–U4. > - The first live L1 waves showed two harness gaps and one missed product shape. > - The pressure writers locked several run rows per statement and deadlocked with product transactions. A real stream does not do that. > - U1 counted an accepted Retry that waits on the system as a live owner forever. Ten tasks sat 8 minutes in that state with no run. > - This pull request fixes both gaps, adds the owed direct test for `collectContentionEvidence`, and records the live wave results. ## Linked Issues or Issue Description Refs #20421, #20385. What happened? The live L1 waves (`w-20261110T040531-s20385` on the #20385 parent, `w-20261110T044338-s20385` and `w-20261110T050818-s20386` on master) produced harness deadlocks (40P01) from multi-row pressure statements. U1 passed tasks whose accepted Retry waited on `process_identity_missing` for 16 minutes without a run. Expected behavior Pressure writers behave like a run-log stream (one row per statement). U1 flags an accepted Retry that waits on the system past the queue bound. Steps to reproduce Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts <results dirs>` over the saved waves, or the L1 wave command in the README. ## What Changed - `contention.ts`: each pressure statement updates one running run row. The old statement updated every running row at once and deadlocked with product transactions. - `universal.ts` (U1): an accepted Retry that waits on the system (`retryWait`) is an owner only for the queue bound (5 min of server-ready time). While it waits, the Retry button, failed notice and blocker do not count as actionable items, because the person already acted. A scheduled retry is still a timer. - `contention-db.test.ts`: a real PostgreSQL test for `collectContentionEvidence` (receipt count, duplicate producer events, terminal receipt, accounting fields, company scope). - README: live wave results table, updated audit numbers and the U1 rule. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 28 files, 345 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Audit over 79 saved waves (1,620 tasks, 1,483 passing): U1 17 flags, none on a passing task or an all-pass wave. The longest accepted-Retry wait on a passing task was 278 s. U2 1, U3 30 (3 on passing tasks, warning only), U4 0. - Live waves (codex and claude, `--workloads L1 --faults K1 --concurrency 4 --tasks 4 --db-pressure 4`): - `w-20261110T040531-s20385`, #20385 parent: 117 receipt-write serialization failures (40001) in the server log; 1 of 4 tasks passed. Three runs failed `adapter_failed` after 40001 exhausted in `createRunUsageRecorder`, `reserveRunBudget` and native finalization. - `w-20261110T044338-s20385`, master `8004ad069e` with the harness: 2 serialization failures; 1 of 4 passed. Failures included 40001 in `createRunUsageRecorder` and deadlocks caused by the old multi-row writers (fixed here). - `w-20261110T050818-s20386`, same build with one-row writers: 0 of 4 passed. Every first run failed in `reserveRunBudget` before spawn, and every accepted Retry then waited on `process_identity_missing` without starting a run. - No wave reached the runner's commit stall, so none showed `native_event_replay_conflict`. ## Risks - U1 can now fail a task whose accepted Retry waits more than 8 minutes. No passing task in 79 saved waves came close (longest 278 s). - One-row writers produce fewer statements per second than the multi-row statement did; `contention-db.test.ts` still reproduces the #20385 receipt race with them. - Test-only change; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 00:22:07 -06:00
DottaandClaude Opus 5.5	2444658c6f	test(restart-chaos): add ST5 credential rejection and ST5R key rotation storage faults (#20455 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` already puts a fault proxy between the server and a loopback MinIO (ST1 outage, ST2 latency, ST3 errors) > - Cloud deployments use the default AWS credential chain, and keys can expire or rotate during a run > - The harness had no fault for a rejected or rotated key, so this failure class was untested > - This pull request adds ST5 (a window of 403 credential rejections from the proxy) and ST5R (a real key rotation at MinIO) > - The benefit is that waves can prove runs wait out credential problems and pick up rotated keys ## Linked Issues or Issue Description What existing behavior does this improve? The restart-chaos storage faults cover outages, latency and 5xx errors only. Subsystem affected tests/restart-chaos (storage chaos). Current behavior No wave can reject the server's S3 credentials or rotate the store's key. Proposed behavior `--storage-faults ST5` answers every request in a window with 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch`. `--storage-faults ST5R` restarts MinIO with a new root key (the old key stops working) and gives the server the new key through its credential source only when the window ends. Reason and benefit The first local ST5 wave (w-20261110T041556-s501) shows the gap: a 41 s `ExpiredToken` window failed 4 of 6 tasks with `adapter_failed`, and the automatic retries failed within one second. The product fix is a separate PR. ## What Changed - `tests/restart-chaos/storage-credentials.ts` (new): ST5 and ST5R rule planning, a wave-owned `AWS_CONFIG_FILE` profile whose `credential_process` prints the current key, MinIO restart with a new root key on the same loopback port, and a proxy wrapper that holds traffic during the restart and publishes the new key when the window ends. - `storage-proxy.ts`: the `expired_credentials` fault takes an S3 error code; a `credential_rotation` marker passes traffic through. - `storage-chaos.ts`, `types.ts`, `config.ts`, `main.ts`: registry lines for ST5/ST5R and the rotation wiring. - README: ST5/ST5R rows and the credential-source description. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (28 files, 328 tests pass). The new `storage-credentials.test.ts` fails before this change (the kinds do not exist) and passes after. It includes a real MinIO test: after a rotation the old key gets 403 `InvalidAccessKeyId`, a client with a cached key fails, and the same client succeeds after it re-resolves its profile. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave: `--env local --workloads A1,C1 --faults K1,K2 --tasks 6 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90` ran and injected 62 credential 403s (storage-proxy.json). ## Risks - Test harness only. No product code changes. - ST5R restarts MinIO. Requests are held for up to 5 s during the restart, so a slow restart can also show as a short 502 blip. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 00:16:28 -06:00
DottaandClaude Opus 5.5	95cb3c94c4	test(restart-chaos): cap store bandwidth during ST2 slow-store windows (#20452 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests how runs survive restarts and cloud storage faults > - A Paperclip-sized remote workspace (R1 on Daytona with `--storage s3`) saves about 730 MiB per snapshot as a multipart upload and restores it as one large download > - The ST2 "slow store" fault only adds latency per request. On a healthy loopback store the whole multipart upload takes about 2 s, so a latency fault adds a fixed delay per part and never makes a large transfer slow > - A real degraded store is slow per byte too. We need that to test large saves and restores that take minutes > - This pull request gives each ST2 window a seeded bandwidth cap that all requests in the window share > - The benefit is that ST2 waves now slow large snapshot uploads and downloads by minutes, and the harness can find save and restore timeouts on big workspaces ## Linked Issues or Issue Description Refs #20299, #20313 (storage outage handling on small workspaces), #20442 (saved workspace Files over 512 MiB). What existing behavior does this improve? The ST2 slow-store fault in the restart-chaos storage proxy. Subsystem affected Test harness (`tests/restart-chaos`). Current behavior ST2 adds 1–10 s of latency before each request. The bytes then flow at loopback speed. A 730 MiB snapshot upload or download is still fast. Proposed behavior ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction is shared by every request in the window, so parallel multipart parts share the cap. Reason and benefit Large saves and restores now take minutes under ST2. This shows the save and restore timing, the storage request counts, and any timeout on a slow store. ## What Changed - New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2 bandwidth draw (2–16 MiB/s). - `storage-proxy.ts`: the `latency` fault takes an optional `bytesPerSecond`. The proxy paces the request body and the response body through one rate per direction for each rule. - `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts keep the rule, so a replay uses the same cap. - README: the ST2 row describes the cap. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/storage-bandwidth.test.ts tests/restart-chaos/storage-proxy.test.ts tests/restart-chaos/storage-chaos.test.ts`: 14 pass. - Red before: with the proxy and ST2 changes reverted, the proxy cap test and the ST2 draw test fail. Green after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. ## Risks - Low risk. Test harness only. ST2 now draws one more seeded number, so a seed gives different later windows than before. Replays read the saved receipts and are not affected. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 00:03:56 -06:00
DottaandClaude Opus 5.5	30d99f4e32	test(restart-chaos): add time faults (TM1-TM3): freeze, sleep with aged rows, aged outage (#20458 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Each wave runs for minutes on a fresh instance. Nothing time-bound ever expires during a wave. > - Production failures on 2026-11-09 were about time: an OAuth refresh after more than 1 h made pending approvals invalid (#20340), and a timeout used the wrong window (#20330). > - This pull request adds a fourth fault family, time faults (TM1–TM3). The faults make leases, deadlines and waits expire during a run or across a restart. They do not change product code. > - The benefit is that waves now find failures that occur only when time passes. The first waves found one: a run whose turn completes after a host sleep fails, and the task needs Retry. ## Linked Issues or Issue Description Refs #20340 Refs #20330 What is the problem? The chaos harness never moves the clock, so it cannot find failures where a lease, a deadline or a pending wait expires. What is the solution? Add three time faults. Each one makes stored deadlines lapse in a way that a real host sees: a process pause, a host sleep, or a long outage. Why is it needed? Two days of chaos testing did not find the failures of #20340 and #20330, because no wave lasted long enough for a deadline to pass. ## What Changed - Add `TM1`, `TM2` and `TM3` to `--faults` (`time-faults.ts`, `time-faults-config.ts`): - TM1 freeze: send `SIGSTOP` to the server process only, for 95–150 s, then send `SIGCONT`. The runner and Postgres continue. The pause is longer than the controller-instance lease (90 s) and the execution-control deadline (60 s). - TM2 sleep: send `SIGSTOP` to the server and all of its descendants. While they are stopped, move every database time back by the age. Then send `SIGCONT`. This is a stand-in for a laptop sleep. - TM3 aged outage: send `SIGKILL` to the server tree (as K1). While it is down, move every database time back by the age. Then start a replacement. With `--workloads G1 --triggers approval`, an approval stays pending for a day across a restart. - Add `--time-age-hours` (1–72, default 26). TM faults run only with `--env local`, and not with storms, crash points or K14. - Aging is one transaction. It moves every `timestamp` column, and every time inside a `jsonb` column (ISO strings in their own format, and epoch milliseconds in the wave's window), back by the same interval. Triggers are off for the shift. Copies of a time in JSON (for example the comment `updatedAt` in a continuation receipt) still match their columns. The first wave showed that a shift of the columns only causes false failures. - The identity, state and time columns of the time-bound tables are written before the shift to `<wave>/time-faults/<faultId>.json`. Per-table counts go to the fault receipt (`timeFault`) and the timeline (`time_fault_aged`). - The observer and grader read through `TimeShiftedApi`. It adds the age back to each time that a shift moved, so R4, R5, U1–U4 and the speed budgets use real time. - Make small additions to the registries: `types.ts` (fault kinds, receipt field, option), `config.ts`, `scheduler.ts` (a seeded freeze length only for TM tasks, so the plans of existing seeds do not change), `supervisor.ts` (the freeze and aging hooks), `main.ts` (one line for the API) and the README. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 30 files, 368 tests pass. The new tests are: - `time-faults.test.ts`: option parsing, freeze order, Postgres exclusion, statement generation, and reading times back at their real time. - `time-faults-db.test.ts`: aging runs on real embedded Postgres. The test checks that trigger rewrites are suppressed, that JSON copies match their columns, that text columns do not change, that the evidence file is written, and that an age the wave nearly outlived is refused. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local, codex), each run from a separate worktree and one wave per slot: \| Wave \| Scenario \| Result \| \| --- \| --- \| --- \| \| `w-20261110T031929-s301` (A1,C1 × TM1–TM3) \| First run, before JSON aging \| t01: TM1 missed, because A1 finished first. t02: pass. t03 (C1): its turn completed after TM2 and failed with `native_session_lease_lost`; a follow-up comment woke a new run. After TM3, that run failed with `continuation_user_authorization_missing` and needed Retry. That second failure was a harness artifact: the receipt kept the comment's `updatedAt` in JSON, which the first version did not shift. JSON aging fixes it. \| \| `w-20261110T041910-s303` (B5 × TM1–TM3) \| Freeze, sleep, aged outage \| t01 and t02 crossed TM1, then TM2. Both turns completed after the thaw and failed with `native_session_lease_lost`, so both needed Retry (R3). This is a product failure; see below. t03 (TM3, 26 h aged outage): pass, with no Retry. \| \| `w-20261110T050125-s302` (G1 × TM3, `--triggers approval`) \| Aged wait: an approval pending for 26 h across a restart \| Pass (2/2). Both approvals were still pending after the restart and the board decided after the fault: t01 rejected, so the task is cancelled; t02 approved, so it is done with exactly one subtask. No Retry; U1–U4 pass. \| \| `w-20261110T051535-s304` (B5, TM2 with `--time-age-hours 1`, then TM1; one task at a time) \| Attribute the failure \| t01 (TM2 sleep, only 1 h): its turn completed after the thaw, then failed with `native_session_lease_lost`, so the task needed Retry. This reproduces the product failure. t02 (TM1 alone, 113 s server-only freeze): pass. The failure follows the database clock jump, not the pause. \| Product failure found (not fixed in this PR). A local native run whose host sleeps longer than the 20-minute execution lease loses its completed work. After the wake, the provider finishes the turn and the brief is saved. Then result acceptance (`native-session-executor.ts`, the transaction at about line 12031) requires `native_run_finalizations.lease_expires_at > clock_timestamp()`. The database clock has moved on, so the run fails with `native_session_lease_lost`, and the task shows "Couldn't finish this attempt … Retry". No other boot claimed the lease: the owner, attempt and controller identity are unchanged, and the server's monotonic deadline did not expire. The lease is fenced by wall-clock expiry alone. A safe fix changes lease fencing semantics, which is a heartbeat-boundary decision, so I report it here rather than include a fix in this harness PR. Reproduce with `--workloads B5 --faults TM2` (`--time-age-hours 1` is enough; see `w-20261110T051535-s304`). ## Risks - Harness-only change; no product code changes. Waves that do not use TM faults are unchanged. `TimeShiftedApi` returns responses unchanged when no shift was applied. - `SIGSTOP` goes only to the wave server's own process tree. The tree is found from the server PID, and Postgres is excluded. Every frozen process gets `SIGCONT` in a `finally` block and in a process `exit` hook. - Aging is a model: the database becomes what it would be if everything happened the age earlier. Times outside Postgres (in-memory timers, runner state files), times inside `text` columns, and HMACs over a time are not moved. The README lists these limits. Agent JWT expiry is not covered: native runs mint a token for each tool call. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 23:44:52 -06:00
DottaandClaude	2c2186706e	test(restart-chaos): human-in-the-loop journeys (Q2, Q3), journey triggers and a journey x fault matrix (#20448 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane while real agents work, and grades each task. > - Almost every harness workload is "the agent does a file task and finishes". Few workloads have a person in the loop. > - On 2026-11-09, production failures came from journeys with a person in the loop: a saved answer that could not reach a dependency-blocked task (#20363), and an app-action approval that a routine OAuth refresh or a one-hour deadline made invalid (#20340). > - No workload exercised these journeys, so no wave could catch them. > - This pull request adds journey workloads with deterministic oracles and journey-specific fault triggers. It also adds a journey × fault matrix and a way to run a wave against an older build. > - The benefit: one command crosses every journey with every fault kind, and the same wave proves red before a fix and green after it. ## Linked Issues or Issue Description Refs #20363, Refs #20340, Refs #20336, Refs #20432 ## What Changed - New workload Q2 (`journeys.ts`, `journey-driver.ts`): the agent saves a question with its native `request_human_input` tool. The board adds an unfinished `backlog` prerequisite, marks the task `blocked` by it, and answers through the task-thread respond route. The oracle requires that the answer reaches the agent in bounded answer mode while the task stays blocked, that the prerequisite relation stays, that no run has `issue_dependencies_blocked`, and that no stale Retry notice stays at rest. Then the board finishes the prerequisite and the agent finishes the task. - New workload Q3: a harness-owned OAuth 2.1 + MCP app on loopback (`journey-oauth-app.ts`, discovery, dynamic client registration, PKCE, rotating refresh tokens, one write tool). The board connects it through the Apps routes and sets the tool to Ask first for a scripted `process` agent. When the review is pending, the harness moves the review's saved timestamps and the token expiry back 61 minutes in the wave's own database, runs Check connection (which refreshes the token through the app), and approves. The oracle requires one review, a refresh between request and decision, an accepted approval, exactly one `tools/call` with the reviewed arguments, an `executed` request, and an agent report that carries a receipt only the provider result contains. - New triggers `asked` and `answered` fire a restart fault 0–3 s after the saved request is first seen and after the board's response. They work for the existing Q1 saved question (#20432) too. The board answers before an `answered` fault fires and after every other planned fault. - New `--matrix` option: one task per workload × adapter × (a `K0` no-fault baseline, each restart fault at each of the workload's own triggers, each provider message fault). `K0` is a new task fault kind with no fault. - New `--server-from <checkout>` option: every server start and restart of the wave runs another prepared checkout, for example the parent of a fix, with the same harness code. - Q3 setup allows unpriced usage on the wave's company budget policy, because a scripted process run records usage with no price and the default policy pauses the whole company. - README documents the journeys, the triggers, the matrix and the red-before-fix recipe. Unit tests cover the oracles (pass and the pre-fix failure signatures), the driver's ordering, the matrix plan, option validation and the OAuth app. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 295 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Live local waves, codex, one wave at a time (shared machine lock), `--concurrency 2`, at most 3 tasks: \| Wave \| Server build \| Command \| Result \| \| --- \| --- \| --- \| --- \| \| `w-20261110T024308-s106` \| this branch (master `5570debc5b` base) \| `--workloads Q3 --faults K1 --matrix --seed 106` \| 3/3 pass (K0, K1^asked, K1^answered); the app saw one refresh before each decision and one `tools/call` \| \| `w-20261110T024447-s106` \| `--server-from` parent of #20340 (`0e325e8d52`) \| same \| 3/3 fail as expected: approve returns 409 `action_expired` ("Tool review has expired"), no `tools/call`, task stays `in_review` \| \| `w-20261110T024644-s107` \| this branch \| `--workloads Q2 --faults K1 --matrix --seed 107` \| K1^asked and K1^answered pass. The K0 cell failed only because the oracle then required a comment; the agent had reported `ANSWER-…` in its bounded turn's run result 14 s after the answer, while blocked. The oracle now accepts either. \| \| `w-20261110T030341-s107` \| `--server-from` parent of #20363 (`eebdfe71b1`) \| `--workloads Q2 --faults K1 --matrix --seed 107` \| 3/3 fail as expected: `answer_reached_agent_while_blocked`. No run starts between the answer and the finished prerequisite (6 minutes of ready server). \| \| `w-20261110T033426-s108` \| this branch, rebased onto #20432 \| `--workloads Q1,Q2 --faults K1 --triggers asked,answered --tasks 4 --seed 108` \| Q1^asked and Q2^asked pass. Q1^answered and Q2^answered fail on a product gap (below). \| \| `w-20261110T020037-s102` \| this branch, before the rebase \| `--workloads Q1,Q2,Q3 --faults K1 --triggers asked,answered --tasks 3 --seed 102` \| Q1 with `asked` and `answered` K1 restarts: both done with the answer. \| Waves `…-s107` (pre-fix) and `…-s106` (pre-fix) used the same harness code as the green waves. Only the server build changed. New product failure (not fixed here). A K1 restart 0.3–3 s after the board's answer can lose the answer turn's native run: - `w-20261110T033426-s108/t04` (Q1, K1 2.9 s after the answer): run `a40cc8f7` resumes with `resume_dead_runner`, then fails `runner_state_identity_mismatch: authority_indeterminate` (`adapter_failed`). The task shows "Couldn't finish this attempt. Your message is saved." and needs Retry. Retry works. - `w-20261110T033426-s108/t03` (Q2, blocked task, K1 0.27 s after the answer): run `5b5355de` is `process_lost`. Its bounded retry `92a57f50` fails `native_session_recovery_failed` (`session.snapshot` failed). The Retry click starts `0e6aa9d1`, which fails `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The task then waits with `retryWait.reason=process_identity_missing` and stays blocked without the answer. - Suspected path: native runner restart reconciliation of the runner state identity (`resume_dead_runner`, journal command settlement) in `server/src/services/native-runtime/` and `packages/paperclip-runner/`. Host load was 8–14 during the wave. ## Risks - Test harness only. No product code changes. - Q2 depends on the model following its instructions. The oracle accepts the answer in either a comment or the bounded turn's run result, because both were seen. - Q3 shifts persisted timestamps only in the wave's isolated database. It does not shift unrelated tables, so it simulates elapsed time for the review and the token, not for the whole instance. - `K0` and the new triggers add values to shared registries (`types.ts`, `scheduler.ts`). Existing plans keep their meaning. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-09 22:37:24 -06:00
DottaandClaude	5ab98b0063	test(restart-chaos): add database contention workload and universal checks U1-U4 (#20421 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks on an isolated control plane and grades them. > - Two production failures on 2026-11-09 passed the harness. Its tasks made little output and little accounting, so database write contention never happened (#20385). Its grading was per workload, so a task stuck for a reason no scenario targeted was not caught (#20361). > - The harness needs a contention workload and checks that apply to every task. > - This pull request adds the L1 workload, a `--db-pressure` knob, and the universal checks U1–U4. > - The benefit is that the #20385 failure class and unexplained stuck or misleading tasks now fail a wave. ## Linked Issues or Issue Description Refs #20385, #20383, #20361. What happened? The harness ran real embedded PostgreSQL, but no task streamed enough run-log output while usage receipts committed. The #20385 receipt race (SQLSTATE 40001, retained outbox, `native_event_replay_conflict`) could not occur. Also, no rule checked every task for an owner or an actionable item, for retry loops, for diagnosable failures, or for stale notices. Expected behavior A chaos wave can create run-row contention between run-log writes and accounting writes and check that accounting stays exact. Every task of every wave gets scenario-independent liveness and notice checks. Steps to reproduce Run an L1 wave with `--db-pressure 4` against the parent of the #20385 merge (red), then against master (green). Commands are in Verification. ## What Changed - `contention.ts` (new): workload L1 (five 3,000-line shell outputs as five tool calls, then a brief), its accounting oracle, and `RunRowPressure`, the `--db-pressure <writers>` knob. The writers run `update heartbeat_runs set updated_at = updated_at` on the wave company's running runs, in the wave's own isolated database only. - L1 oracle: no `native_event_replay_conflict` or `runner_event_commit_stalled`; each producer event journaled once; a terminal receipt for each succeeded native run; finished runs accounted; per-run cost equals its cost events; the board's spend equals the cost-event sum with no runs pending accounting; and proof that the task produced contention. - `universal.ts` (new): U1 liveness, U2 no silent retry loop, U3 diagnosable failures, U4 no stale notices. `grader.ts` grades them on every task. U1, U2 and U4 fail a task. U3 is a warning. - `invariants.ts`: `readUniversalDbEvidence` reads failed-run error text and recovery-action causes from the wave database at wave end, for U3. - `universal-audit.ts` (new): runs U1–U4 over saved wave evidence to measure false positives and bound headroom. - `contention-db.test.ts` (new): on real PostgreSQL, under the pressure writers, the pre-#20385 repeatable-read receipt write exhausts its retries with 40001 and the current writer commits. - Small registry edits: `types.ts` (L1 id, `dbPressure`, optional `contention` and `universal` fields), `workloads.ts` (L1 entry), `config.ts` (`--db-pressure`, L1 not a default), `main.ts` (start and stop the writers, read database evidence before teardown), `ledger.ts` (universal verdicts and warnings in the ledger and `SUMMARY.md`). - README: "Database contention" and "Universal checks" sections, with the audit table and bound reasons. ## Verification - `contention-db.test.ts` (real embedded PostgreSQL): with `--db-pressure` writers (4), the pre-#20385 repeatable-read receipt write exhausted its retries with 40001 in 8 of 8 attempts; the current READ COMMITTED `persistUsageReceipt` committed 8 of 8. This is the harness-level red/green for #20385. - Live L1 waves (`--env local --adapters codex,claude --workloads L1 --faults K1 --concurrency 4 --tasks 4 --seed 20385 --db-pressure 4`), red on the #20385 parent `bf818280f2` with this harness overlaid and green on this branch: queued behind the shared one-wave-at-a-time lock and not yet run. The first red attempt failed at startup (the fresh checkout lacked a built plugin SDK; no tasks ran). Live results are not claimed here. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: all harness unit tests pass, including the new `universal.test.ts`, `contention.test.ts` and `contention-db.test.ts`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Universal-check audit over 74 saved waves (1,592 tasks): U1 7 flags (one C1 stall and six R1 runs queued 8 min behind a retry row, the #20361 shape), U2 1, U3 22, U4 0. None of U1, U2 or U4 flags a passing task or an all-pass wave. U3 flags 3 passing tasks (all `adapter_failed`), so it ships as a warning. ## Risks - U1, U2 and U4 can now fail a task that passed R1–R6 and T. The audit found no such case in 70 saved waves. The bounds are environment-configurable. - `--db-pressure` is opt-in and writes no values. It writes only to the wave's own temporary database. - Other agents edit the same harness. This change keeps registry edits small; `grader.ts` and `invariants.ts` changes are additive. - No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-09 22:02:57 -06:00
DottaandClaude Opus 5.5	0c7a628fd1	test(chaos): read remote R1 files from the execution workspace and record why a read failed (#20440 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades Daytona R1 tasks by reading their files through `GET /issues/:id/file-resources/content` > - In replays w-20261110T010031-s9913 and w-20261110T012332-s9950 the tasks recovered (R1, R3, R6 pass), but all 506 content reads returned 422, even after the closing save left the workspace clean (`snapshotId` set, no writer, `location: null`) > - The harness discarded each error, so the waves could not show which 422 the server returned > - This pull request asks for the execution workspace explicitly and records the last error per file in the task record and the timeline > - The benefit is that the next wave names the exact cause instead of a blind `module_present` failure ## Linked Issues or Issue Description Follow-up to #20348. What happened? Every file read for a finished Daytona R1 task returned 422, and the response was discarded. Expected behavior Failed reads leave their reason in the evidence. Actual behavior Only the outcome, a `null` file, was kept. ## What Changed - File reads add `workspace=execution`, so the server reads the issue's execution workspace and not the `auto` choice. - `readRemoteWorkspaceFile` and `captureRemoteWorkspaceFiles` report each failure through a callback. The observer keeps the last message per file in `record.remoteFileErrors` and logs `remote_file_read_failed` once per distinct message. - The remote-workspace file reader stays in `remote-workspace-files.ts`; `observer.ts` is at 993 lines. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 263 passed. The test server now requires `workspace=execution`, so the existing remote tests were red before, and a new test records the 422 `workspace_unavailable` message. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - If a Daytona task's files only exist in a project workspace rather than its execution workspace, an explicit `execution` selector reports that clearly instead of falling back. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 21:34:35 -06:00
DottaandClaude	1b52f359dd	test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 What existing behavior does this improve? The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. Subsystem affected `tests/restart-chaos/` (test harness only). No product code changes. Current behavior No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. Proposed behavior `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - CF2 red before #20361 (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-09 21:10:42 -06:00
DottaandClaude	4158799d1b	test(chaos): add provider message faults (PV1-PV3) and a saved-question scenario (#20432 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos`) tests that agent work survives faults. > - The harness has two fault families: control-plane restarts (K) and storage faults (ST). > - On 2026-11-09, two production failures came from a third family. The provider was alive, but a protocol message was late or missing (#20330, #20336). > - The harness could not produce that family, so it did not catch these failures. > - This pull request adds provider message faults (PV1 delay, PV2 drop, PV3 duplicate) on the real path between the runner and the Codex app-server. It also adds a saved-question scenario (Q1) that uses the faults. > - The benefit: the harness can now find this class of failure, and it fails against both pre-fix commits. ## Linked Issues or Issue Description Refs #20330 Refs #20336 What happened? A Codex interruption terminal that arrived after 18 s failed a governed wait (#20330). A dropped terminal after a saved handoff failed shutdown settlement with "Couldn't finish this attempt" (#20336). The chaos harness could not inject either fault. Expected behavior The harness can delay, drop or duplicate a chosen provider protocol message for one task. A scenario fails on the pre-fix product code and passes on master. Steps to reproduce Run the wave commands in Verification. Run them on master and in a worktree at the pre-fix commit with `tests/restart-chaos` copied from this branch. ## What Changed - `provider-faults.ts` and `provider-shim/`: the PV1–PV3 fault family. - A resolve hook goes into the wave's server through `NODE_OPTIONS`. It answers only the `@openai/codex/package.json` lookup that product code uses to find the local Codex. It returns a stand-in package in the instance temp root, and then removes itself from `NODE_OPTIONS`. - The stand-in `codex` bin is a stdio shim. It starts the real Codex with the same arguments and relays JSON lines unchanged. It changes only the message that the task's rule names: the `turn/completed` after a `turn/interrupt` (`terminal`) or the response to `turn/interrupt` (`interrupt_ack`). Each rule applies once per task. The task nonce binds the rule to the task. - Every decision goes to `<wave>/provider-faults.jsonl`. A PV task fails `provider_fault_applied` when its fault did not happen. - There is no product change. Thus the same harness runs against older product commits. - `question-handoff.ts`: Q1 workload. The agent saves one question with `request_human_input`. After the asking run settles, the board answers through the respond route. The oracle checks that the question was asked once, the asking run succeeded, no run failed, nothing ran before the answer, and the brief has the answer. - PV faults run only with `--env local --adapters codex`, only with workloads that interrupt a turn (Q1, S1, G1), and not in the same wave as restart faults. A PV1 delay passes only when the held message was released. - Registry edits: `--faults` accepts PV1–PV3. New options are `--provider-fault-message` and `--provider-fault-delays` (a sweep, cycled across tasks). The edits are in `types.ts`, `config.ts`, `scheduler.ts`, `workloads.ts`, `observer.ts` (Q1 board step) and `main.ts` (install the seam and add the check). - README section with commands, the seam, and what is not covered. ## Verification - Unit tests: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`. 23 files and 279 tests pass on the final head. The new tests drive the real shim against a stand-in provider for each kind (delay, drop, duplicate, interrupt ack, once per task, held message discarded on exit, a multibyte character split across writes). They also run the resolve hook in a child Node process. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (real Runner, real Codex 0.160.0, gpt-6.1-sol, macOS arm64): Pre-fix runs use a worktree at the merge parent with `tests/restart-chaos` copied from this branch. Common arguments: `--env local --adapters codex --workloads Q1`. \| Scenario \| Product code \| Wave \| Result \| \| --- \| --- \| --- \| --- \| \| 1. PV1, terminal 18 s late (`--faults PV1 --provider-fault-delays 18 --tasks 1 --seed 101`) \| master (`8004ad069e`) \| `w-20261110T000333-s101` \| pass: terminal held 18 s and released, asking run `succeeded`, one question, brief has the answer \| \| 1. same \| `34c1e6469c` (parent of #20330 merge) \| `w-20261110T014734-s101` \| fail: Codex stopped at 14.9 s with the terminal still held, asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt. Your message is saved.", Retry needed (R3) \| \| 1. sweep 5/14/25/29 s (`--faults PV1 --provider-fault-delays 5,14,25,29 --concurrency 2 --tasks 4 --seed 102`) \| master \| `w-20261110T015426-s102` \| 4/4 pass \| \| 1. same sweep \| `34c1e6469c` \| `w-20261110T015708-s102` \| 5 s and 14 s pass; 25 s and 29 s fail (`native_provider_terminal_failed`, Retry needed): the old 15 s drain \| \| 2. PV2, terminal dropped (`--faults PV2 --tasks 1 --seed 103`) \| master \| `w-20261110T015239-s103` \| pass: terminal dropped, asking run `succeeded` through verified shutdown, one question, nothing ran before the answer \| \| 2. same \| `a67b936885` (parent of #20336 merge) \| `w-20261110T015010-s103` \| fail: asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt", Retry needed (R3) \| The first master wave exited 1 for one cold `agent_boot` speed sample (8.6 s against a 7 s p95 budget), before any fault, while a preflight build ran in parallel. Its task passed. No new product failure was found on master. ## Risks - Test-only. No product code changes. - The shim adds one Node process in front of Codex for PV waves only. Other waves do not change. - The nonce binds a rule to a Codex process. If one Codex process serves two tasks, the second task's traffic can bind it too. Q1 tasks use separate sessions. - Claude is not covered. Claude runs through the ACPX sidecar against a hash-verified provider pack, and its ACP protocol has no interrupt acknowledgement. A Claude shim needs a different seam. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use, code edits and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-09 20:31:18 -06:00
DottaandClaude Opus 5.5	490d97bf1e	test(perf): guard Mine against waiting on the company-wide workspace list (#20419 ) > Follows #20388 (merged as `50df9a19b9`). This PR adds only test and fixture files. It changes no product code. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Mine inbox (`/:companyPrefix/issues?view=mine`) is the first page an operator opens > - On an instance with 8,758 execution workspaces, Mine took about 13 s to show rows. It loaded every workspace in the company (`execution-workspaces?summary=true`, 2.5 MB) and kept all rows hidden until that read finished > - #20388 fixes this. Mine now reads only the workspaces its rows reference (`ids` filter), and waits for them only when the rows show workspace data > - The collections perf suite did not catch the regression. Its fixture has almost no execution workspaces, and it has no check on the workspace read > - This pull request adds a workspace-scale fixture and a spec that fails if Mine requests the company-wide list again, or if Mine rows wait for the workspace read when no workspace data is shown > - The benefit is a regression guard that fails on `master` before #20388 and passes with it ## Linked Issues or Issue Description Refs #20388 What happened? The collections perf suite could not detect that Mine waited on a company-wide execution-workspace list. The fixture had no workspace scale, and no assertion covered the workspace request or the reveal gate. What did you expect? A browser check that fails when Mine requests the unfiltered workspace list, or when Mine rows wait on a workspace read that does not change what the rows show. Steps to reproduce Run `tests/perf/collections` against `master` UI assets on a company with thousands of execution workspaces. All existing checks pass, although Mine requests the full list and holds rows for it. ## What Changed - `tests/perf/collections/workspace-fixture.ts` (new): a retained company with 6,000 execution workspaces. 4,000 of them belong to finished tasks. 12 Mine tasks each run in their own isolated workspace. Mine tasks come from the API. The API cannot create execution workspaces, so the bulk rows go to the disposable instance database directly (same pattern as `tests/e2e/legacy-failure-continuation.spec.ts`). The database URL comes from `PAPERCLIP_CONFIG` or `PAPERCLIP_COLLECTION_PERF_DATABASE_URL`, loopback only. The fixture uses the existing retain/404 rule. - `tests/perf/collections/mine-workspaces.spec.ts` (new), cold load of `/:prefix/issues?view=mine`: 1. Mine requests only `ids`-filtered `execution-workspaces?summary=true` reads, and only for the workspaces its rows reference. No unfiltered read. 2. With the Workspace column off and no workspace grouping, all Mine rows appear while every workspace read is held. 3. With the Workspace column on, each row shows its isolated workspace name. 4. Records cold time from navigation start to the first Mine row (two frames after mount, as in `measure-mine-context.ts`) over at least 5 runs. It attaches the samples and the median. It is not a budget. - `tests/perf/collections/README.md`: documents the fixture, its database prerequisite, and the checks. ## Verification Local disposable instance (`onboard --yes --run`, `local_trusted`, embedded PostgreSQL, `PAPERCLIP_UI_DEV_MIDDLEWARE=false`, production UI builds). Both arms used the same server (this branch) and the same database. Only the UI assets changed, so this is a UI comparison. The `master` UI does not send `ids`. ```sh PAPERCLIP_CONFIG=<instance>/config.json PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:3219 \ pnpm exec playwright test --config tests/perf/collections/playwright.config.ts mine-workspaces.spec.ts ``` Fails on `origin/master` UI at `5858cef8f9` (before #20388 merged): ``` ✘ 1 Mine reads only the execution workspaces its rows reference Error: Mine requested the company-wide list of 6000 workspaces + "/api/companies/<id>/execution-workspaces?summary=true", ✘ 2 Mine rows appear while workspace reads are pending when no workspace data is shown Error: Mine rows waited for the workspace read expect(locator).toBeVisible() failed … timeout 15000ms ✓ 3 the Workspace column resolves each Mine row's isolated workspace name ✓ 4 records cold time to Mine rows 2 failed, 2 passed ``` Passes on this branch (#20388 UI): ``` ✓ 1 Mine reads only the execution workspaces its rows reference ✓ 2 Mine rows appear while workspace reads are pending when no workspace data is shown ✓ 3 the Workspace column resolves each Mine row's isolated workspace name ✓ 4 records cold time to Mine rows 4 passed ``` Cold time to first Mine row, 5 runs each, fresh browser contexts, 1440×900: \| Scenario \| `master` UI median (samples) \| This branch median (samples) \| \| --- \| --- \| --- \| \| Native loopback, no interception \| 472 ms (461, 466, 472, 483, 523) \| 425 ms (406, 417, 425, 429, 998) \| \| Workspace reads delayed 3,000 ms (diagnostic, intercepted) \| 3,523 ms (3488–3539) \| 546 ms (471–1754) \| On this fixture the unfiltered list is 1.55 MB but returns in about 40 ms on loopback. The filtered read is 3.2 KB in about 4 ms. So native loopback timing does not separate the arms. The delayed scenario shows the dependency: on `master`, rows wait for the full workspace delay. No hard time budget is added. The suite has no CI timing convention, and the structural assertions are the guard. CI status of this suite: `tests/perf/collections` runs in no GitHub workflow. It is a manual suite against a disposable local instance (see its README). The new spec runs the same way as the existing specs: same directory, same `playwright.config.ts`, no new config. The Vitest checks in #20388 (`ui/src/pages/Inbox.test.tsx`) remain the CI guard for the reveal rule. Full collections suite on this branch: 19 passed, 5 failed. The 5 failures are existing cases (`startup-regressions`: two partial-source cases and two 429 cases; `startup`: "Coordinator Inbox retains aggregate Mine reads when Streamlined UI is disabled"). They fail the same way on the `origin/master` UI with the new spec excluded. This PR does not cause them. `pnpm verify`: exit 2 (README is a global CI input), so required CI supplies broad coverage. `tsc -p tests/tsconfig.playwright.json` passes (the CI typecheck project that covers `tests/perf`). ## Risks - Low risk. Test and fixture files only. - The fixture writes rows straight to the database. It needs `PAPERCLIP_CONFIG` or `PAPERCLIP_COLLECTION_PERF_DATABASE_URL`, and refuses non-loopback databases. Without either variable the spec fails with a clear message. - The timing case changes the real `enableIsolatedWorkspaces` instance setting and restores it in `finally`. - Seeding 6,000 workspaces takes about 10 s on the first run. Later runs reuse the retained fixture. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, Playwright runs). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 20:29:56 -06:00
Nicky LeachandClaude Opus 5.5	ff4ba7d319	perf(runner): run launch artifacts in place from a trusted read-only root (#20428 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description What existing behavior does this improve? Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. Subsystem affected Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). Current behavior Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. Proposed behavior When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. Reason and benefit See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - Root checks at startup. The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - Per-artifact checks, on the open descriptor. After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - Admitted artifacts become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3\|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - This change needs review from a security owner before merge. It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - Misconfiguration fails closed. A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - Residual trust. On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - Allowlist. Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 17:46:33 -08:00
DottaandClaude Opus 5.5	b184e40108	perf(ui): serve every project list from one includeArchived read (#20395 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI reads the company project list in many places: the sidebar, starred projects, the command palette, the new-task dialog, and archive-aware pages such as the Mine inbox > - Active-only consumers read `GET /projects`, and archive-aware ones read `GET /projects?includeArchived=true`, under two React Query keys > - A Mine load therefore fetched the same large payload twice (580–590 KB each on our dev instance) > - The two responses differ only by archived rows; the server's active list is exactly the rows where `archivedAt` is null > - This pull request makes every consumer share the `includeArchived=true` query, and active-only consumers filter it with `select` > - The benefit is one project read per page instead of two ## Linked Issues or Issue Description What existing behavior does this improve? Duplicate project list reads on page load. Subsystem affected Board UI project queries (`ui/src/lib/company-projects-query.ts` and its consumers). Current behavior A cold Mine load sends both `/companies/:id/projects` (sidebar) and `/companies/:id/projects?includeArchived=true` (inbox). Proposed behavior One `includeArchived=true` read serves both. Active-only consumers apply `selectActiveProjects`. Reason and benefit The project list is one of the largest shell payloads. Reading it once saves a large response on every page that has both kinds of consumer. ## What Changed - New `companyProjectsQuery(companyId, { includeArchived? })` in `ui/src/lib/company-projects-query.ts`. It always uses the `includeArchived: true` key and fetch. For active-only use it adds a stable, module-level `select` that keeps rows without `archivedAt`. - 22 files that read the active list now spread `companyProjectsQuery(...)`, and so do the plugin SDK project picker and 16 archive-aware pages that wrote the shared key and fetcher inline. Their `enabled` and other options are unchanged. `OnboardingWizard`'s one imperative read is unchanged. `pages/Inbox.tsx` already reads the same key and fetcher, so it shares the entry without an edit. - Invalidation: project mutations and live events invalidate `queryKeys.projects.all(companyId)`, which matches the shared key. The onboarding wizard invalidated the old active-only key; it now invalidates `projects.all`. - Tests: new `company-projects-query.test.tsx` (one request for both consumers, correct filtering, and one refetch after `projects.all` invalidation). Three existing tests (`CommandPalette`, `CompanyExport`, `GitHubBotManagement`) now seed or expect the shared key. - New browser check `tests/perf/collections/shared-project-list.spec.ts`: after the Mine inbox read completes, the page has sent no active-only project request. ## Verification - `pnpm verify` passes. - `ui`: the consumer suites (`CoordinatorSidebar`, `SidebarProjects`, `SidebarStarredProjects`, `CommandPalette`, `NewIssueDialog`, `AgentWorkPanels`, `CoordinatorNeedsYou`, `IssueDetail`, `ProjectDetail`, `Projects`, `CompanyExport`, `ChatExecutionDefaults`, `GatewaysList`, `Search`, `IssueProperties`, `persisted-query-cache`) pass. - Browser check: `PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/collections/playwright.config.ts shared-project-list.spec.ts`. It passes on this branch and fails on `master`. ## Risks - A page that needs only active projects now downloads archived rows too. For most companies that is a small difference; the duplicate read it removes is a full copy. - The server does not order either list, and consumers sort for display, so filtering on the client keeps behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser). 1M-token context window. Extended thinking was off. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 17:42:47 -06:00

1 2 3 4 5 ...