216 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	433d94ee0b	refactor(maps): generate indexes from local story and feature metadata (#16214) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Feature recipes and Storybook help contributors find behavior, examples, and verification steps. > - Independent feature changes currently update shared catalogs, indexes, and review hashes. > - Those shared files cause merge conflicts and repeated bookkeeping during parallel work. > - This pull request stores metadata in each feature recipe and story, then generates the combined views. > - Contributors can merge independent work without editing a common index. Git and PR evidence retain change and review history. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintaining the product, agent-access, and Storybook maps while multiple agents change different features. **Current behavior** A story or recipe change can update `catalog.json`, `INDEX.md`, review hashes, verification tables, and tracked generated navigation. Two unrelated changes then conflict in the same files. **Proposed behavior** Each story declares capability links and classification in `parameters.paperclip`. Each feature recipe declares its navigation and optional verification guidance. Startup, build, and UI typechecking discover these sources and generate ignored indexes and Storybook pages. **Reason and benefit** A routine change edits its own source files. Aggregate metadata is rebuilt from the merged checkout. There is no sidecar registry, shared review journal, or hash acknowledgment to maintain. **Breaking changes** The tracked Storybook catalog and index paths are retired. Consumers use `ui/storybook/generated/` or `maps/generated/` after generation. Contributor instructions and the existing feature-gap browser suite use the new paths. Story and overview IDs remain stable. ## What Changed - Move the existing annotations into 262 story files. Preserve classifications, capability associations, exports, and explicit IDs. - Discover all 35 feature recipes. Store section/order beside each recipe and move durable verification routes out of the shared table. - Remove tracked catalogs, inventories, overview pages, and inline preview ordering. Keep authored preview behavior in `preview-base.tsx`. - Generate the Storybook guide, feature/access Markdown indexes, gap report, page-reference inventory, and preview wrapper automatically. - Update the map contracts and maintenance skills to use Git history and revision-specific PR/CI evidence. - Add schema, identity, regeneration, and actual Git merge tests. Run these through the existing UI Vitest suite. Add focused browser navigation checks. This is one coordinated migration. Most of the 367 changed files contain only moved metadata or deleted generated content. Review the generator scripts, map contracts, Storybook configuration, and tests first. No product API or authorization behavior changes. ## Verification Passed: - `node scripts/check-maps.mjs`: 35 paired areas and 175 capabilities. - `node scripts/storybook-catalog.mjs --check`: 262 suites and 2,485 authored stories after incorporating current master. - `node --test scripts/check-maps.test.mjs scripts/storybook-catalog.test.mjs scripts/storybook-feature-index.test.mjs`: 28 tests, including independent branches merged in both orders with identical generated output and no tracked changes. - `pnpm --filter @paperclipai/ui exec vitest run storybook/map-tooling.test.ts`: the CI entrypoint runs those map tests. - `pnpm check:token-gates`. - `pnpm -r typecheck` and `pnpm build` before the final master rebase. - `pnpm build-storybook`: all 2,517 original story/docs URLs retained. Current master contributes exactly four additional recovery stories. - Focused Playwright navigation checks: search and open a feature without examples; expand verification guidance and check source links; retain capability order and follow a related Foundations suite. The final clean Storybook build passed after deleting every ignored map output. Post-rebase UI typechecking and both browser tests passed. The working tree stays clean after generation. The concurrent-cleanup regression fails with `ENOENT` without the fix and passes with it. All GitHub CI gates passed on `00fc7ec18e`, which also received Greptile 5/5 with no unresolved findings. The final merge commit `1c3c5143a6` incorporates master and resolves only the two modify/delete conflicts by retaining removal of the aggregate index and catalog. Master’s new workspace feature and access guidance is preserved. Map validation, catalog generation, overview generation, all 28 map tests, and diff checks pass after conflict resolution. The maintainer explicitly requested merging without another Greptile review. CI has not been rerun to completion on this merge commit. The local repository-wide `pnpm test:run` was stopped before completion after the earlier CI matrix passed; it is not reported as a local pass. No screenshot baselines or production acceptance are claimed. To repeat the browser check against an owned Storybook server: ```sh PAPERCLIP_GAP_STORYBOOK_URL=http://127.0.0.1:6136 pnpm exec playwright test --config tests/storybook-visual/feature-gap-stories.config.ts map-navigation.spec.ts ``` ## Risks - Existing branches need a one-time rebase to move their catalog changes into local annotations. The PR incorporates the recovery changes from current master. - Changing the same story or capability can still produce a meaningful conflict. This removes the shared aggregate bottleneck. - The generator uses Storybook's existing CSF parser. Literal-only annotations and parser regression tests protect the current contract. - Generated Markdown is available after running the generator; the published Storybook provides a browsable guide. - Shared source changes still require review. Removing hashes removes mechanical acknowledgments; it does not establish runtime parity. ## Model Used OpenAI Codex, based on GPT-6, with repository inspection, code execution, and browser-test tools. The exact deployment model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the previously reviewed head `00fc7ec18e`; final conflict-resolution checks are documented above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups on `00fc7ec18e`; another review was waived by the maintainer - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-13 15:27:23 -05:00
Devin FoleyandPaperclip	5a6d0d08a7	fix: record verified workspace file recovery from tasks (#16104) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must preserve files and show the next operator action. > - A stopped run can leave a file repair hold after its recovery action is resolved. > - The task notice shows that hold but has no form to record verified file recovery. > - A chat connection can also replace the file repair instruction with a chat retry instruction. > - This pull request adds a file recovery form and preserves the file repair instruction. > - The form records evidence for the original run and keeps the current task status. ## Linked Issues or Issue Description **What happened?** A retained workspace file hold can block a task after its recovery action is resolved. The task page offers no file recovery form. A chat binding can hide the required file repair instruction. **Expected behavior** An authorized board member can record verified recovery of the original files. The server checks the original stop proof. The task keeps its current status and does not start another run. Chat instructions must not replace file repair instructions. **Steps to reproduce** 1. Create a stopped legacy run with a retained workspace recovery marker and a resolved recovery action. 2. Open its task page, including a task with a chat connection. 3. Observe that the file hold has no supported form to record verified recovery. Refs: #15857, #15824, #16095. Those PRs address shutdown export, native finalization and workspace source isolation. This PR uses the existing file repair API. ## What Changed - Add a board-only file recovery form with evidence and an explicit file verification confirmation. - Preserve the current task status when recording file recovery. - Project whether the original recovery marker supports file recording. - Keep file repair instructions when a task has a chat binding. - Add UI, route and Storybook examples for success, denial and chat-connected file holds. - Document the recovery flow and update the affected Storybook catalog entries. ## Verification - Merged master `c8efdbfba43` and resolved both rounds of catalog/index conflicts. Upstream review notes and classifications are preserved, together with all four retained-file recovery examples and their stable story IDs. Both fix branches also combine without merge conflicts or catalog fingerprint drift. - Passed locally after conflict resolution: 47 tests covering the file recovery UI, execution blocker notice and retained workspace repair route. The earlier ownership-route pass also covered 196 tests. - Passed: UI token gates, the complete Storybook catalog check with no drift errors, feature and agent map structure checks, and all 17 catalog/index script tests. Verification inventory counts match all 262 authored suites and 2,485 story IDs. The upstream production verification obligations remain intact. - Passed: full workspace typecheck, build and Storybook build on latest head `10ac22b35d0a4fe469cf620914845c509046da48`. The built index retains all four file-recovery story IDs. - Latest head `10ac22b35d0a4fe469cf620914845c509046da48` has 54 successful checks and two intentionally skipped Storybook checks. Apex is 5/5 on that exact commit, with no unresolved findings. The full general/serialized suites and all eight browser shards passed in CI. Local tests use the focused suites above. Storybook browser interactions have not been executed locally. - Manual check: open a task with a retained file hold. Verify that evidence and confirmation are required. Record verified file recovery. Confirm that its status stays unchanged and no run starts. Repeat without board runtime access and with an unverified stop receipt. ## Risks - The form records an operator attestation. It does not recover files itself or prove external action outcomes. - A stale source or missing stop proof must remain rejected by the existing server route. - No database migration or automatic task retry is added. ## Model Used OpenAI GPT-6 (`gpt-6`), used through Codex with reasoning, repository tools, code editing and test execution. The exact context window size is not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-13 13:47:10 -05:00
DottaandPaperclip	e67db42dc9	docs: define heartbeat execution design principles (#16190) ... ## Thinking Path > - Paperclip coordinates AI agents and their work. > - Heartbeat controls when that work starts, who owns execution, and how it settles. > - The extracted modules need shared architectural guidance as they evolve. > - Rules should protect those boundaries without freezing the current implementation. > - This PR adds scoped heartbeat principles for contributors and reviewers. > - The rule helps future changes preserve execution authority and recoverable work. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** The heartbeat service has a module map but no scoped architectural rule under `rules/`. **What's wrong?** Changes span admission, execution, completion, and recovery. Reviewers need shared principles for authority, ownership, and settlement across these boundaries. **Suggested fix** Add a concise heartbeat rule, route contributors to it from `AGENTS.md`, and register the same scope for Greptile. Keep implementation details in the code and existing developer documentation. Related: #16172 established scoped repository rules. #15984 completed the latest heartbeat extraction. Searches found no duplicate heartbeat-guidelines PR. ## What Changed - Add `rules/heartbeat-execution-boundaries.mdc` with a folder guide, a concise map of existing module responsibilities, and six design principles. - Keep `heartbeat.ts` focused on coordination: extend an existing owner or extract cohesive modules when new behavior would substantially grow it, with tests and explicit contracts. - Cover current execution authority, explicit ownership, distinct completion obligations, recovery intent, durable state, and visible uncertainty. - Register matching paths in `.greptile/files.json` and link the rule from `AGENTS.md` and `doc/DEVELOPING.md`. - Include behavioral review criteria and flag substantial coordinator growth or mixed responsibilities while allowing justified cohesion and ordering exceptions. ## Verification - Parsed the rule YAML and Greptile JSON. Confirmed that the rule globs exactly match the Greptile scope. - Checked 14 positive and negative scope cases with Node's `path.matchesGlob`, including nested heartbeat modules, sibling services, regression tests, and unrelated paths. - Verified local links and heading anchors, including the existing heartbeat module map, and confirmed all 20 named modules exist. Reviewed their responsibility descriptions against the current code and developer documentation. - `git diff origin/master --check` passed after rebasing onto current master. - Reviewed the principles against the current heartbeat code and tests. The diff changes four documentation and review-configuration files. - Application typecheck, runtime tests, and build were not run locally. This worktree has no installed dependencies. CI and review status are tracked below. - CI passed on `a17f5a2cc82b7bb7e4629d4a2d9e3a766e27c5c0`: 55 successful checks, two skipped, no failures or pending checks. This includes typecheck, build, server/runner tests, and browser tests. - Greptile reviewed the same commit at 5/5 with no findings or unresolved review threads. ## Risks The rule can influence later implementation and review decisions. It applies to changed heartbeat behavior and permits justified contract changes with tests. The scoped globs exclude unrelated subsystems. This PR does not change application behavior. Hosted loading of the new rule has not been demonstrated by the local configuration checks. ## Model Used OpenAI GPT-6 Astra (`gpt-6-astra`) via Codex, with repository inspection, file editing, and command execution. The context-window size was not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (scoped rule/configuration checks described above) - [x] I have added or updated tests where applicable (documentation/configuration validation; no runtime changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-13 10:48:17 -05:00
3 people	9bd820e687	feat(storybook): mirror the complete feature-map index (#16052) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The feature map describes its user-facing capabilities and verification recipes. > - Storybook provides isolated UI examples for those capabilities. > - The previous organization showed only features with primary story suites. > - This pull request derives the complete Storybook index from the upstream map. > - Readers can find the same features in both places and see related examples or explicit gaps. ## Linked Issues or Issue Description Refs #16046 and #16049. **What existing behavior does this improve?** Storybook navigation and the downstream feature catalog. **Current behavior** The sidebar has 28 primary feature homes. The feature map lists 35. The main guide groups suites alphabetically by area slug. Related examples can live under a different feature. **Proposed behavior** Use the feature map's eight sections and 35 features in the same order. Give each feature an Overview page with all its capabilities, related example links, and missing-example messages. Keep shared foundations outside the product index. Preserve all existing stories and URLs. **Reason and benefit** Readers can move between the feature map and Storybook without learning a second index. A feature without cataloged examples remains visible. **Breaking changes** None to existing story IDs or product behavior. Storybook start and build now reject stale generated feature pages and ordering. Run the catalog generator after upstream changes, inspect the output, and review affected source fingerprints. ## What Changed - Generate 35 feature Overview stories and capability data from the upstream map. Generate the inline sidebar order required by Storybook. - Replace the main guide with the full ordered feature index and search. Link all related suites from feature pages, even across primary homes. - Preserve all 223 authored files, 2,210 story IDs, and the existing docs ID. Keep status, implementation, and fixture classifications visible. - Put the generated Markdown catalog in feature-map order. Keep generated guides outside example counts. - Update maintenance and review skills, documentation, and generator tests. Keep the feature map independent of Storybook. ## Verification - Passed: 16 catalog and generator tests; map checks; token gates; both skill validators. - Passed: `pnpm -r typecheck`, `pnpm build`, and the Storybook build. - Built-index comparison preserved all 2,211 previous entries and added only 35 feature overviews. Feature order matches the map. - Browser: inspected the complete index and sidebar, searched for Goals, opened its overview, and followed a related suite under Projects. The original suite renders. - Passed: all 8,170 UI tests in 728 files. Browser checks also covered empty search, capability-ID search, missing-example pages, the original editable-text bookmark, the CLI sidebar label, and searching/opening shared foundations. - The broad local `pnpm test:run` was stopped after a failure in the unchanged GitHub chat test `authorizes description edits as the editor, never inheriting the original author's access`. That exact test passed alone without code or expectation changes. A complete local repository suite pass is not established. - Original revision CI passed after one runner retry. The merged revision `5ca01c70716f23974680a8019ddc613035d11bc4` passed fresh CI. Local UI typecheck, catalog checks, 16 generator tests, map checks and token gates pass. - Greptile: 5/5 on final commit `5ca01c70716f23974680a8019ddc613035d11bc4`, with all findings resolved. - Master merge: inspected source drift in 61 catalog entries and regenerated INDEX, preserving upstream notes. Marked the sidebar suite that mounts IssueDetail unreviewed for the new activity query; recorded limits for composer and channel-default changes. No stories removed. - Final Storybook build passed. Browser filename searches for `new-task-composer` and `forms-editors` return the expected feature and shared example. - No visual-baseline or full production acceptance claim. The deeper authored scenario groups remain intact. ## Risks - The parser depends on the feature-map Markdown contract. It rejects unknown index rows, omitted feature recipes, and unexpected generated files. - A suite can reference several capabilities. A link proves catalog association, not complete behavior coverage or production parity. - Storybook treats `/` as a folder separator. Sidebar labels use a comma for that character; the guide and page headings retain the upstream name. - No database, API, authorization, or production component changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser inspection. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>	2026-10-13 07:01:29 -05:00
3 people	56a0bf0d49	Support test Cloud connector identities (#16033) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its Cloud connector uses an environment-bound identity to call a hosted broker. > - The identity parser and enrollment flow accept development, staging, and production. > - A separate test broker needs its own environment identity. > - This pull request accepts test identities and maps the test broker to that environment. > - Known hosted brokers still reject identities from other environments. ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud connector configuration and self-host enrollment. **Current behavior** The app rejects an identity with environment `test`. Self-host enrollment infers development for the test broker. **Proposed behavior** Accept `test` identities at `https://test.paperclip.app`. Infer test during self-host enrollment. Reject development, staging, and production identities at that broker. Continue to reject test identities at the production and staging brokers. **Reason and benefit** Branch testing needs a separate broker so test identities and requests stay in their selected environment. The existing identity, enrollment, and signature flow can provide that isolation with a new allowed environment value. **Breaking changes** Existing environment values and broker checks keep their current behavior. Older app versions cannot use test connector identities. ## What Changed - Add `test` to managed and self-hosted connector environment types and validation, and to the board API enrollment status type. - Infer test from its hosted broker and enforce the matching environment. - Test enrollment persistence, signed request environment/audience claims, and broker mismatch rejection. - Document test broker configuration in the development guide. ## Verification - Connector, enrollment, and board API-client suites passed: 63 tests across three files. Board UI typecheck and token gates passed after synchronizing its enrollment response type. - `pnpm -r typecheck` and `pnpm build` passed, including after rebasing onto current master. - `pnpm test:run` completed its general server stage: 17,867 passed, 105 skipped, and 9 failed across 5 files. The runner stops at that failing stage, so it did not run subsequent workspace/serialized stages locally. All nine failures also reproduce on the unchanged base commit in a separate checkout. Five skill-related cases resolve an unrelated ancestor skills directory; two workspace cases accept a remote main branch that their fixture expects to be absent; two permission-cleanup cases return macOS `ENOTEMPTY` instead of the expected permission error. - Current-head GitHub CI passed, including full general and serialized test suites, typecheck, build, runner checks, and browser e2e. Greptile is 5/5 with no unresolved comments. - `git diff --check` and Gitleaks passed. ## Risks The new environment value is additive. The test broker must support test identities before use. This change preserves existing signature handling, managed identity overrides, and rejection of mismatched identities at known hosted brokers. A local full-suite pass is not claimed; the nine failures above also occur without this change. ## Model Used OpenAI Codex, GPT-6, with reasoning and tool execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <34892728+cryppadotta@users.noreply.github.com>	2026-10-12 19:32:33 -07:00
DottaandPaperclip	f329b24ee1	feat: give tasks durable project-optional workspaces (#15749) ... Give tasks durable company-scoped workspace bindings independent of project assignment, channel defaults and authorized repository preparation. Reuse verified input seeds and sparse output recovery while preserving trust, privacy and controller provisioning boundaries. Full CI and E2E, Superagent, legacy checks and Greptile Apex 5/5 passed on reviewed head 753680b61c. All review findings are resolved; the PR records focused verification and historical acceptance limits. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-12 20:45:18 -05:00
DottaandPaperclip	88bf4c6c1d	refactor(storybook): organize folders and navigation by feature map (#16049) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The feature map describes product areas and verification entry points. > - Storybook has 223 story suites that show production components, proposals, and retained history. > - The parent PR maps those suites to features, but their folders and sidebar still use unrelated groups. > - This pull request puts the existing suites under feature folders and uses the feature map's names and order in the sidebar. > - Explicit meta IDs preserve existing story links while the titles change. > - Contributors can find stories through the same product structure without losing examples. ## Linked Issues or Issue Description Refs #16046. This is the requested follow-up. Parent #16046 is merged. This PR now targets master, with its single follow-up commit rebased onto the parent squash merge. The Storybook implementation is unchanged by that rebase. Related open Storybook PRs were checked; this change organizes existing suites and adds no product capability. ## What Changed - Move all 223 story files into folders named for their primary feature area. Keep useful scenario groups below the feature's sidebar label. - Order the sidebar by the eight feature-map sections. Put Overview first and shared Foundations last. Leave areas with no stories out of the sidebar. - Move 19 support files into `ui/storybook/support/` and update relative imports and active documentation. - Preserve all 2,210 story IDs and the single docs ID through explicit CSF meta IDs. Keep all entry names and story exports. - Record stable IDs and sidebar titles in the downstream catalog. Validate feature folders, title prefixes, and IDs. Update the maintenance skill and review evidence. ## Verification - Storybook build passed. Built index comparison against parent `75bd3c977cd828d5222823850339e8204f40eb47` found the same 2,211 entries, IDs, and names. - UI typecheck passed (`ui/node_modules/.bin/tsc -b` from `ui`, invoked as `./node_modules/.bin/tsc -b`). - UI Vitest project: 728 files and 8,176 tests passed. - Catalog checker and its nine tests passed. Feature-map checks, token gates, and the maintenance-skill validator passed. - Chromium: nine existing play cases and one render case reached `storyFinished: success` with no page exceptions. The guide rendered, sidebar sections followed map order, and a search for `steering/queue` opened its preserved story link. - Source comparison found only meta title/ID changes, rebased relative references, and updated fixture path text in story files. - To inspect: build Storybook, start `node scripts/serve-storybook-static.mjs --port 6136`, and open the Overview guide. Expand feature folders and use existing story links. - The complete repository suite was not repeated for this organization change. The parent records a backend timeout that passed in isolation, without a complete broad-suite pass. Visual comparisons remain blocked by the unpublished empty baseline. The separate fixture-server story was not run. ## Risks - Open branches that add stories under old paths will need to choose a feature folder and update the catalog. Cross-feature suites have one primary home and retain related capability references. - Sidebar paths change; story and docs IDs remain stable. Automated references to source paths may need the new location. - This is one requested follow-up with 258 changed files, mostly moves. It exceeds the usual review-size guideline because splitting it would leave the sidebar and folder convention partly migrated. - Browser verification samples ten stories, not all 2,210. Existing production/prototype classifications and review gaps remain unchanged. - Required CI gates and a 5/5 review passed on the original head. Fresh checks and review are running for the rebased head. Catalog validation and all nine checker tests passed after rebase; UI, map, skill, and checker contents are identical to the original reviewed head. ## Model Used - OpenAI Codex, GPT-6. The exact runtime model ID and context-window size are not exposed in this session. Used reasoning, repository inspection, code execution, and local browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 22:29:22 +00:00
DottaandPaperclip	9e721264e9	feat: maintain Storybook through a downstream feature catalog (#16046) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Storybook supports review of its production UI and proposed designs. > - Its 223 story files have no consistent link to product capabilities or record of component and mock provenance. > - Old compositions can drift while they still look like current product examples. > - This pull request adds two maintenance skills and a catalog that consumes the existing feature map. > - The first audit preserves all stories and fixes proven fixture drift. > - Contributors can now find examples by feature and see what still needs review. ## Linked Issues or Issue Description **What existing behavior does this improve?** Storybook maintenance, feature discovery, and the accuracy of isolated UI examples. **Current behavior** Stories are already centralized under `ui/storybook/`, but their sidebar has 51 top-level groups. Production components, local prototypes, and historical designs are not consistently distinguished. The old guide uses manual coverage claims. GitHub searches found no duplicate feature-map catalog change. **Proposed behavior** Keep existing story locations and IDs. Add a downstream catalog keyed to stable feature capabilities. Record product status, rendered subject, data boundary, and declared review dependencies separately. Use dedicated maintenance and review skills to reconcile drift and report evidence. **Reason and benefit** The catalog makes existing examples reusable without treating mock behavior or old designs as production acceptance. The feature map remains independent of Storybook. ## What Changed - Add `maintain-storybook` and `review-storybook` skills with contributor guidance. - Inventory all 223 story files. Generate a feature index and provide searchable navigation in the existing Overview story. Track direct subjects through relative imports and local wrappers, including the primitives demonstrated by foundation suites. - Add a checker for inventory, capability references, duplicate IDs, and declared-source drift, with eight regression tests. The parser test covers real nested CSF files, export filters, explicit IDs, and play counts. - Preserve historical designs and classify mixed wrappers. Record the initial audit and remaining parity work. - Align the question-dismiss/send fixture with production behavior. Add dismissal and confirmation-reopen regressions and repair the editor selector. - Exclude three exported fast-response fixtures from story discovery and add the missing render-time callback spy. ## Verification - Passed: `node scripts/storybook-catalog.mjs`, `node --test scripts/storybook-catalog.test.mjs` (8 tests), and `node scripts/check-maps.mjs`. - Passed: validators for both new skills, `pnpm check:token-gates`, `pnpm -r typecheck`, and `pnpm build`. - Passed: final Storybook build; nine selected browser `play` cases; three fast-response render cases; feature-guide search/link navigation; question-fixture reset on reload. - Built-index comparison preserved all 2,208 original story IDs and added only two new regression cases. - Passed: `pnpm exec vitest run --project @paperclipai/ui` (728 files, 8,176 tests). - The full `pnpm test:run` has reported a 15-second timeout in the untouched backend case `connects advertised Workspace gmail.read without mutating definitions in authenticated`. The exact test passed standalone without changing code, expectations, or timeouts. The remaining broad run was stopped after this failure. The full suite is not verified. - Visual comparison is blocked by the existing empty baseline manifest. The managed-accounts API/DB fixture was not started. See `maps/storybook-map/REVIEW.md` for exact scope. ## Risks - This is an initial inventory and targeted audit. 188 suites still require deeper state and fixture parity review. - Review hashes cover declared dependencies. They do not follow all transitive production imports or prove semantic correctness. - The catalog requires upkeep when stories, mapped recipes, or declared subjects change. Shared mocks still need feature-by-feature contract review. - No production application source, database schema, or upstream feature-map contract changes. ## Model Used OpenAI GPT-6 via Codex. Used reasoning, shell tools, file editing, and Chromium browser verification. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 17:20:00 -05:00
DottaandPaperclip	f5346c05b9	ci: route PR admission through reserved AWS capacity (#16042) ... ## Thinking Path > - Paperclip uses CI to verify changes before release. > - Trusted PR tests can run on isolated AWS workers. > - Their runner-selection job currently waits for GitHub-hosted capacity. > - A large hosted test queue therefore blocks admission to AWS. > - A dedicated routing workflow lets us reserve capacity for metadata validation. > - This PR moves the routing job to reserved AWS capacity after the bootstrap workflow and pool are available. ## Linked Issues or Issue Description **Issue type** CI reliability enhancement. **Where is the issue?** The Select trusted runner job in PR CI. **What's wrong?** The routing job competes with a large hosted test queue. Its sampled median wait was about 20 minutes. AWS tests cannot begin before this job finishes. **Suggested fix** Add a protected reusable routing workflow. Deploy its dedicated AWS pool, then switch the PR caller in a follow-up PR. No overlapping open PR or issue was found. ## What Changed - Call ci-routing.yml on protected master from pr-trusted.yml. - Check the canonical repository and approved PR author/sender before allocating reserved AWS metadata capacity. Unapproved PRs continue to use hosted routing. - Permit approved manual routing canaries on protected master. - Use a validated roster snapshot for placement. Invalid JSON, missing settings, stale snapshots, and unapproved identities select hosted routing without a placement-time JSON parse. - Preserve the full in-job identity, rerun actor, repository, head, merge, and stack checks. - Update stack-scope and placement regression tests and document the pool and rollback. ## Verification - The routing scripts and placement expression tests passed (44 tests), routing-plus-shard tests passed (56 tests), and the existing .github/scripts suite passed (381 tests). - Existing routing allow/deny scenarios passed with the extracted script (16 cases). - actionlint passed with the dedicated custom runner label declared in its local configuration. - git diff --check passed. - Full pnpm typecheck/test/build cannot run in this worktree because dependencies are not installed. The change is limited to workflow YAML and Node tests. - The dedicated pool is deployed with two warm On-Demand workers and a four-worker ceiling. The routing qualification passed: it started about five seconds after workflow creation, selected AWS, and finished while the hosted gate in the same PR remained queued. - Bootstrap https://github.com/paperclipai/paperclip/pull/16039 is merged. The deployed group now authorizes only ci-routing.yml on protected master, and its master canary passed. ## Risks - The maintainer explicitly approved a queued-CI admin bypass after current-head Greptile 5/5 and focused verification. - Activation requires the deployed routing Fleet and its group to authorize the routing workflow on protected master. Keep the bootstrap dependency explicit. - The group must authorize only ci-routing.yml on protected master and canonical repository ID 1170821064. - The routing workflow runs no checkout or PR code and has read-only permissions. The test admission checks are unchanged. ## Model Used - OpenAI GPT-6 via Codex, with reasoning, shell tools, and file editing. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused routing and workflow tests; full pnpm commands are blocked by missing worktree dependencies) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] Required CI status checked; the maintainer explicitly approved admin bypass for queued CI after clean current-head review and focused verification - [x] Greptile is 5/5 on 993fbe695a with no open findings or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 16:47:57 -05:00
DottaandPaperclip	f0e12c8519	Add managed Slack setup with an independent own-app path (#15679) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Slack connects a person to an agent through an installed app and a verified identity. > - Independent installations must remain usable without Paperclip Cloud or manager credentials. > - Approved Cloud deployments can use Slack manager grants to create and install each agent's app. > - Both paths should share credential storage, identity checks, recovery, and the existing runtime. > - This pull request adds gated managed setup and keeps independent app setup under Advanced. > - It also makes ongoing connection settings and personal test messages easier to use. ## Linked Issues or Issue Description **Subsystem affected** Slack setup, connection settings, tenant credential storage, and the Cloud connector handoff. **Problem or motivation** Approved deployments need an installation flow without pasted configuration tokens. Open-source users must retain their independent Slack app flow. A connection manager also needs a simple way to check a bot from Settings. **Proposed solution** Use the managed path when the instance is configured to support it. Choose agent and Connect are the two visible stops. Continue starts authorization or installation. Use your own app remains under Advanced before creation. Save personal manager grants by company, Paperclip user, Slack user, workspace, and manager app. Reuse only verified identities for test messages. Unknown identities must complete a private Slack connection challenge before linking and receiving the test DM. **Alternatives considered** A shared bot would change each agent's Slack identity. Requiring Slack CLI or a hosted service for self-hosted users would remove independent installation. **Roadmap alignment** This maintainer-requested change improves Connected Apps and experimental Agent Chat. It adds provisioning and management controls to the existing Slack runtime. **Additional context** The companion Cloud broker profile is a separate deployment prerequisite. Managed setup remains gated. Existing-app adoption is deferred. ## What Changed - Extract Slack registration, provider requests, completion, customer-owned setup, managed setup, and personal manager grants into focused services. Preserve the compatibility facade and independent removal cleanup. - Add company-scoped manager grants and a registration reference through an additive, idempotent migration after current master. Keep runtime credentials separate from app-registration secrets. - Add authenticated manager authorization, sealed handoff, serialized refresh and revocation, app creation, managed installation, and installer-bound OAuth fallback. - Preserve creation intent, block automatic uncertain recreation, save credentials before downstream checks, and validate app, workspace, bot, scopes, and installer identity. - Add the two-stop production wizard and ordered stories. Keep the own-app, manual, existing-app, and legacy-draft paths. - Simplify Settings. Open Slack configuration targets the saved app. The test-message action has its own row. Remove the old manual-avatar section, channel-message section, and broken permissions link. - Add paginated account discovery and a private connection challenge for unknown accounts. Expired invitations stop waiting and offer a new private invitation. Save send receipts before provider dispatch and preserve confirmed delivery when activity logging fails. - Reinstall the saved managed app on explicit reconnect, replace revoked credentials, and resume completion. Persist reconnect intent and duplicate-request receipts without creating another app. - Update provider browser tests, recovery tests, secret redaction, setup documentation, and the agent prompt. Preserve current master's durable verification scheduling. ## Verification - Full workspace typecheck and repository build passed on `56d389d47a`. The subsequent reconnect retry fix in `288364eada` also passed UI typecheck and UI build. - 114 focused Slack registration cases and 13 production managed-wizard cases passed, including fresh-token reconnect, retry after installation failure, and duplicate reconnect requests. - All nine affected Slack browser cases passed against the production UI and a disposable fixture server. They cover managed setup, automatic setup, own-app selection, refresh, recovery, mobile behavior, and connection settings. - Eight focused real-database test-message cases passed. They cover a private ownership challenge, saved grant reuse, account pagination, revoked and conflicting links, workspace mismatch, disabled DMs, revoked membership, audit failure, ambiguous dispatch, and secret canaries. - UI token gates and the production Storybook build passed. - Greptile reviewed exact head `288364eada5d9c114fad991ff634612639d06672` at 5/5 with no actionable findings or unresolved review threads. - The full repository test suite and current-head CI are running. Their final results will replace this line before merge. - Earlier live feature previews created two bots in the authorized Paperclip Slack workspace and reused one personal manager grant. Local managed Continue created and installed a bot without a configuration token or a separate Add to Slack action. Avatars, welcome DMs, signed callbacks, and direct replies were observed. A Settings test DM also arrived in the signed-in user's Slack account. - Earlier staging thread qualification required manual publication recovery for one bot. Current-head live qualification remains required before enabling the production default. Fixture results do not establish Slack approval behavior in other workspaces. ## Risks - Personal manager tokens can create apps. They stay in scoped vault entries. The manager client secret stays at the broker. Company, actor/session, origin, and app identity checks apply throughout setup. - The additive migration must also work on qualification instances that applied earlier branch migrations. Table, column, index, and constraint creation is idempotent. - Creation and message timeouts can leave uncertain outcomes. Saved request IDs prevent automatic duplicate dispatch. - Unknown Slack identities must receive and complete their private challenge while signed in. Account selection alone does not grant Paperclip permissions. Existing revoked or conflicting links are not overwritten. - A development-only manager transport uses an explicitly configured private credential file for local qualification. Production ignores this transport; no manager credentials are bundled with open source. - Managed rollout requires Slack approval, the companion broker deployment, configured secrets, and final live qualification. Removing a connection does not delete its Slack app. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser tools. The session does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 15:34:02 -05:00
DottaandPaperclip	7d9b93aee2	docs: establish paired feature and agent access maps (#16025) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors use the feature map to understand and verify product capabilities. > - Agents need APIs and usable entry points within their permission boundaries. > - Product recipes and agent access need separate records with shared capability IDs. > - This pull request establishes that contract and gives maintenance ownership of both maps. > - The benefit is repeatable access review without changing product behavior or permissions. ## Linked Issues or Issue Description Refs: #15983, which added the portable verification skills and updated the feature map. This is its contract-only follow-up. Related: #15288 added the original feature map. **Issue type** Missing documentation and maintenance contract. **Where is the issue?** The product map, its missing agent access companions, and the verification skills. **What's wrong?** There is no shared convention for API presence, agent entry points, permission boundaries, access gaps, or evidence. An API declaration does not prove that a permitted native or remote agent can use it. **Suggested fix** Keep the product and agent access maps as siblings under `maps/`. Reuse capability IDs. Record representative source inventory separately from operation review and live proof. Keep both maps aligned through the maintenance skill. ## What Changed - Move the existing product map to `maps/feature-map/`. Preserve its recipes and Poteto format. Update relative links and add companion links. - Add `maps/agent-access-map/` with 35 area files and 171 matching capability IDs. Seed representative API operations, agent paths, policies and known gaps from the reviewed source inventory. Mark entries as `baseline`, `source-only`, and live verification `not-run`. - Define the shared contract in `maps/README.md`. Track API presence, agent availability, policy, evidence and review depth separately. Require operation enumeration before marking a broad capability operation-reviewed. - Extend `maintain-verification-skill` to reconcile both maps and inspect actual route policy and production tool admission. Update creation, verification, notices and contributor references to use the shared contract. - Add `scripts/check-maps.mjs` and ten regression tests. Check matching areas/IDs, indexes, references, contract fields, API method/path examples, justified exceptions and evidence-state consistency. - Keep the coverage inventory unchanged. Preserve skill invocation policies and both MIT license blocks. No product API, permission, UI, runtime helper, migration, dependency or workflow changes. ## Verification - Passed: `node scripts/check-maps.mjs` reports 35 paired areas and 171 matching capability IDs. All local map references resolve. - Passed: `node --test scripts/check-maps.test.mjs` runs ten tests. Fixtures cover missing companions, renamed/orphan/duplicate IDs, broken references, missing operation fields, empty exceptions, missing API examples, empty policy fields, and verified evidence without passed verification. - Passed: metadata validation for all three verification skills. Skill and contributor links resolve. Invocation policies and both license blocks are unchanged. - Compared relocated recipes with current master. Contents differ only in relative links and companion links. `coverage.json` is byte-for-byte unchanged. Master's updated read-only interaction test reference is retained. - Passed: `git diff --check`. The branch starts from current master and contains only this contract change. - Repository-wide typecheck, tests and build were not run locally for this documentation and Node structural-check scope. Live agent access was not exercised. Those gaps are explicit in the baseline entries; the structural check does not certify behavior. CI and automated review are pending. ## Risks - The companions are representative source baselines. They are not complete operation inventories or passed agent journeys. - The checker cannot establish actual permissions, native tool admission, operation completeness or truthful live evidence. Maintenance must inspect source and verify with the actual permitted agent context. - External links to the old product-map directory may need updates. Repository references have been updated. - Existing access gaps and board/user restrictions remain recorded. This contract does not change authority or implement missing paths. ## Model Used OpenAI GPT-6 through Codex assisted with this change. The session does not expose the exact serving model ID, configured context-window size, or reasoning setting. Capabilities used: source analysis, file edits, shell commands, and structural regression tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 14:52:28 -05:00
3 people	7df517d18f	perf(build): reuse a current Runner build in the server's prepare:runner-vendor step (#15926) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Its CI runs a `Build` job on every pull request. That job runs `pnpm build` and then verifies the public npm install. Since the server test shards got faster, `Build` is the slowest check in most runs (419-497s on the six fully green `master` runs created 2026-10-11 19:46Z to 20:17Z). > - Inside `pnpm build`, the server package is the last package to build. Its first step, `prepare:runner-vendor`, runs the full build of `@paperclipai/paperclip-runner` so the server can vendor `packages/paperclip-runner/dist`. In `pnpm -r build` that package was built seconds earlier by the same command. The second build took 12-16s on those six runs and sits on the critical path of the job. > - The step cannot just be removed. A standalone `pnpm --filter @paperclipai/server build` (the Dockerfile's path) must produce a current vendored Runner on its own. > - This pull request keeps that guarantee and skips only the build that is provably redundant. The Runner's `build` records a digest of its inputs, captured before its first step and confirmed after its last, together with a manifest of every file it left under `dist`. The server's `prepare:runner-vendor` reuses the build when the digest matches, the stamp is for this platform, and every recorded output is still there at its recorded size. Any other state runs the same full Runner build as before. > - The benefit is 12-16s off the `Build` job and off every other `pnpm build` (the Canary Dry Run's `release.sh` build, local builds), with no change to what a standalone server build produces. The PR's own first run measured it: the server step reused the build in 56ms where `master` runs rebuilt for 11-15s. ## Linked Issues or Issue Description No GitHub issue exists for this. The description below follows the enhancement template. **What existing behavior does this improve?** The server build's `prepare:runner-vendor` step, which builds `@paperclipai/paperclip-runner` so the server can vendor its `dist`. **Subsystem affected** Build and CI (`server/package.json`, `packages/paperclip-runner/package.json`, the `Build` job). **Current behavior** `prepare:runner-vendor` runs the Runner's full `build` every time. Inside `pnpm -r build`, the Runner was already built by the same command 20 to 25 seconds earlier, with every check passing. On six fully green `master` runs (2026-10-11, 19:46Z to 20:17Z) the repeated build took 11.8s to 15.9s, measured from the `server build$ pnpm run prepare:runner-vendor` log line to the Runner's `Finished release profile` line. The server package builds last, so that time is on the critical path of `Build`, which is the slowest check in most pull request runs. **Proposed behavior** - The Runner's `build` starts with `node scripts/record-build-inputs.mjs --begin` and ends with `node scripts/record-build-inputs.mjs --finish`. `--begin` removes the previous stamp and records a digest of every build input before the first step reads anything. `--finish`, after every step and check has passed, recomputes the digest and writes `packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json` only if the inputs are still the ones the build began with, together with the path and size of every file under `dist`. A stamp therefore exists only for a build that completed every step on inputs that did not move while it ran. - The server's `prepare:runner-vendor` becomes `node scripts/prepare-runner-vendor.mjs`. It reuses the build if the stamp is for this platform and architecture, every recorded output is still there at its recorded size, and the recomputed digest matches. In every other case it removes the stamp and runs `pnpm --filter @paperclipai/paperclip-runner build`, the exact command the step ran before. - `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` forces the rebuild. **Reason and benefit** 12-16s off the `Build` job and off every other `pnpm build` (the Canary Dry Run's `release.sh` build, local builds), with no change to what a standalone `pnpm --filter @paperclipai/server build` produces. **Breaking changes** None. The rebuild path runs the same command as before. A fresh checkout, a changed input, a missing or truncated output, an edit that landed during the build, or a stamp from another platform always rebuilds. ## What Changed - `packages/paperclip-runner/scripts/build-inputs.mjs` (new): defines the input set, the digest, the output manifest and the reuse decision. Inputs are every file under `packages/paperclip-runner` except `dist`, `dist-browser`, `runner/target`, `node_modules` and dot-directories; every file under `packages/paperclip-eval-kernel` except `dist` and `node_modules` (the Runner's `build:typescript` builds it first); every file under the root `patches` directory (`check:acpx-profiles` hashes `acpx@0.13.1.patch`, and `build-verified-provider-entrypoints.mjs` copies the `pi-acp` and `brace-expansion` patches into `dist/cli/pi-provision-inputs`); and the root `package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and `tsconfig.base.json`. The toolchain pins are inside the package (`rust-toolchain.toml`, `runner/Cargo.lock`) and the TypeScript version is in the lockfile, so both are covered. A symlink is hashed as its target path and is not followed. Outputs are every file under `dist`, the tree the server copies wholesale, recorded with their sizes. Exports `beginRunnerBuild`, `recordRunnerBuild` and `assessRunnerBuild` (returns `current` plus a stable reason code: `no_stamp`, `schema_mismatch`, `platform_mismatch`, `missing_output`, `outputs_changed`, `inputs_changed`, `inputs_unchanged`). - `packages/paperclip-runner/scripts/record-build-inputs.mjs` (new): the two phases, `--begin` and `--finish`, added as the first and last step of the Runner's `build` in `packages/paperclip-runner/package.json`. Each build chain has its own pending record under `node_modules/.cache/paperclip-runner-build-inputs.pending/`, named after the shell pnpm runs the chain in (the parent of both phases; `--finish` also accepts a record named after its own pid, for a shell that execs the last command of the chain), so two Runner builds running at once in one checkout do not consume each other's record. `--finish` without a pending record for its chain is an error, so the chain cannot silently lose its first step; `--finish` after an edit that landed mid-build prints `not recorded: inputs changed while the build ran` and leaves no stamp. Records of builds that never finished are pruned by the next `--begin` after a day. - `server/scripts/prepare-runner-vendor.mjs` (new): the reuse-or-rebuild step. It resolves the pnpm that is running it through `npm_execpath` with the same rule as the CLI's `resolvePnpmInstallInvocation` (a JavaScript entry runs through this Node, a native executable runs directly, otherwise `pnpm` on PATH). It prints one `paperclip.runner-vendor.timing.v1` line with `action`, `reason`, `check_ms` and `build_ms`, so the saving is measured on this PR's own `Build` job. - `server/package.json`: `prepare:runner-vendor` now runs that script. The `build` and `typecheck` scripts are unchanged. - `server/src/__tests__/server-package-build-script.test.ts`: pins the new indirection and the command behind it (imported from the script, so the Dockerfile's standalone path stays pinned), and a new test asserts the Runner's `build` starts with `--begin`, ends with `--finish`, and has exactly those two record steps. - Tests: `packages/paperclip-runner/scripts/build-inputs.test.mjs` (node:test, added to `test:typescript:prep`) and `server/scripts/prepare-runner-vendor.test.mjs` (vitest, in the server lane). - `doc/DEVELOPING.md`: a "Reused Runner build in the server build" section under Test Commands. ## Review findings on the first head, all fixed in [2d57542](https://github.com/paperclipai/paperclip/pull/15926/commits/2d575425977f87b4612222d0a1961bc9cc5ba569) Greptile scored the first head 2/5 with three P1 findings. All three were valid. | Finding | Fix | | --- | --- | | **Patch edits leave stale output.** The root patch files the build reads were not in the digest. | Every file under the root `patches` directory is an input. Tests assert the patches are in the input list and that editing the pi-acp patch, the acpx patch, or adding a patch changes the digest. | | **Missing helpers pass the build.** Only four outputs were checked, so a `dist` that lost `dist/cli/acpx-runtime-sidecar.cjs` still counted as current. | `--finish` records the path and size of every file under `dist`; reuse requires all of them intact and reports `outputs_changed` naming the first missing or resized file. Schema bumped to `paperclip.runner-build-inputs.v2` so an older stamp is rejected. Tests cover a deleted helper, a truncated helper, and the server step rebuilding on `outputs_changed`. | | **New edits count as compiled.** The digest was hashed after the build, so a source edited while `build:binary` ran was recorded as compiled. | The build is bracketed: `--begin` captures the digest before the first step, `--finish` compares after the last and writes nothing if it moved. Tests cover the mid-build edit, the undisturbed build, and the CLI in both orders. | ## Review finding on the second head, fixed in the third commit Greptile scored the second head 4/5 with one P1, valid: two Runner builds running at once in one checkout (server `build` beside `typecheck`) shared one pending record, so the second `--begin` overwrote it, the first `--finish` consumed it, and the second `--finish` threw `no pending build inputs` after every step of its build had passed. Reproduced on the second head's module: two `beginRunnerBuild`, then the first `recordRunnerBuild` records and the second throws. | Finding | Fix | | --- | --- | | **Overlapping builds fail each other.** One pending file for every build chain. | One record per chain, named after the shell pnpm runs the chain in: `--begin` sees it as its parent, and so does `--finish`, unless the shell execs the last command of the chain (bash can), in which case `--finish` is that shell and finds the record under its own pid. Pending records of builds that never finished are pruned after a day. An edit between two overlapping starts leaves no stamp whichever build finishes last: the build that began before the edit reports `inputs_changed_during_build` and removes the stamp the other wrote, so a stamp always means every build that wrote `dist` read the inputs it names. Tests: both finishing orders with and without an edit, the pruning, a `--finish` that cannot use another chain's record, and two real `sh -c` chains (plus one under `bash`) held open at the same time, both recording. On the real repo, two `pnpm --filter @paperclipai/paperclip-runner build` chains started 8s apart with a cargo stand-in both recorded (`files=1223 outputs=1434`) and left the pending directory empty. | ## Measured on this PR's runs The `Build` job ran `pnpm build` from the branch, so both builds of the Runner in that job used the new chain. First head (c23faab, run 38175281058), against the four fully green `master` runs created 20:41Z to 21:37Z the same evening: | | PR run 38175281058 | `master`, four runs | | --- | --- | --- | | Runner build inside `pnpm -r build` | 55.3s | 47.3-62.0s | | Server step `prepare:runner-vendor` | `action:reuse,reason:inputs_unchanged,check_ms:56,build_ms:0` | full Runner rebuild, 11.0-14.8s | | Server package build | 29.7s | 32.3-43.5s | | `pnpm build` step | 114s | 102-135s | | `Build` job | 411s | 409-474s | Second head (2d57542, run 38178686999), against the two fully green `master` runs created in the same half hour (38178951607 and 38180574768): | | PR run 38178686999 | `master`, two runs | | --- | --- | --- | | Runner build inside `pnpm -r build` | 53.9s (`begin:` 22:03:04, `recorded:` 22:03:57, digest 460e02616081 over 1231 files and 1434 outputs) | 62.2s, 64.6s | | Server step `prepare:runner-vendor` | `action:reuse,reason:inputs_unchanged,check_ms:109,build_ms:0` | full Runner rebuild, 14.5s, 15.2s | | Server package build | 27.4s | 43.5s, 44.8s | | `pnpm build` step | 106s | 91s, 95s to the Runner's end plus the rest; 127s, 135s in all | | `Build` job | 405s | 461s, 479s | The rebuild is gone from the server build on both heads, and the check costs 56-109ms where `master` rebuilds for 11-15s; the rest of the spread between runs is runner speed on the `runs-on` fleet. The third commit changes nothing on the reuse path: it only changes where the pending record lives between `--begin` and `--finish`. ## Verification - `node --test packages/paperclip-runner/scripts/build-inputs.test.mjs`: 11 tests, three consecutive runs clean. Two builds at once in one checkout each keep their own pending record and both record, in either finishing order; with an edit between their starts the build that began before it leaves no stamp whichever finishes last. A build's start prunes records older than a day and no others. A `--finish` cannot use another chain's record. Two chains run by separate `sh -c` shells at the same time (and one under `bash`) both record and leave the pending directory empty. The input set is exactly the sources, the eval kernel, the root patches and the root config, and skips outputs and caches. The output manifest lists every file under `dist` with its size and nothing outside it. The digest is stable across reads, does not change when `dist`, `dist-browser`, `node_modules`, `.vite` or `runner/target` change, and does change for a Runner source, a Rust source, an eval-kernel source, the pi-acp patch, the acpx patch, a new patch, the lockfile, the base tsconfig, a new file and a rename. A symlink changes the digest but its target's files are not read. A build is current only with a matching v2 stamp, the same platform and architecture, the four named outputs present, and every recorded output intact (a deleted or truncated helper is `outputs_changed`; a file the build did not produce may appear). An input edited between `--begin` and `--finish` leaves no stamp and removes the pending record; the next build records normally. `--finish` without `--begin`, or after a build that left a required output missing, throws and leaves no stamp. The CLI brackets the build for the package it is pointed at, reports a mid-build edit, and rejects `--finish` first, both flags, neither flag, and unknown arguments. - `vitest run scripts/prepare-runner-vendor.test.mjs src/__tests__/server-package-build-script.test.ts` in `server/`: 21 tests. The decision reuses a current build, rebuilds a stale one, and rebuilds a current one only when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1`. The pnpm resolver covers a JavaScript entry, a native executable, a Windows executable, no `npm_execpath`, and npm's `npm_execpath`. With an injected build command: a current stamp runs nothing and prints `action:reuse`; changed inputs run the exact command from `server/` with the stale stamp already removed; a lost `dist/cli/acpx-runtime-sidecar.cjs` rebuilds with `reason:outputs_changed`; a failing build returns its status and leaves no stamp. End to end, the CLI with a stand-in `pnpm` on PATH that brackets its build the way the real one does rebuilds on the first call, reuses on the second, rebuilds after a source change, rebuilds after a lost output, and reuses again. The guard test asserts `prepare:runner-vendor` is the new script, the rebuild command is still `pnpm --filter @paperclipai/paperclip-runner build`, the server build still starts with the step, and the Runner's `build` starts with `--begin` and ends with `--finish` with exactly two record steps; it fails against the previous build script on its first assertion. - Real repo, this checkout, with `cargo` and `pnpm` stand-ins on PATH (no Rust toolchain here; the stand-in writes the one file `stage-runner-binary.mjs` copies): `pnpm --filter @paperclipai/paperclip-runner build` ran the complete chain through the real nested pnpm (every TypeScript step and check, the stand-in cargo, the binary staging, the two binary-reading checks) and recorded `files=1223 bytes=18735059 outputs=1434`; the stamp is 128 KB. `pnpm run prepare:runner-vendor` in `server/` then printed `action:reuse,reason:inputs_unchanged,check_ms:222,build_ms:0`. Appending a newline to `patches/pi-acp@0.0.33.patch` made the assessment `inputs_changed`; truncating or deleting `dist/cli/acpx-runtime-sidecar.cjs` or `dist/cli/opencode-app-server-proxy.cjs` made it `outputs_changed` in 1-5ms; restoring each made it current again. `--begin`, an edit to `src/index.ts`, then `--finish` printed `not recorded: inputs changed while the build ran` and left no stamp. - The whole check on the real set (1223 inputs, 18 MB; 1434 outputs) takes 165-227ms on this host and took 56ms on the CI runner; the output manifest part is 1-5ms. - Not run here: the real `cargo build`, and `pnpm build` end to end. This PR's `Build` job runs `pnpm build` from the branch, so its log carries the Runner's `begin:` and `recorded:` lines and the server build's `paperclip.runner-vendor.timing.v1` line with the measured result. ## Risks - A build input outside the digest would let a stale Runner be vendored. The digest covers the whole Runner package except its outputs, the eval kernel the Runner builds first, the root patches, and the root files the Runner's tsconfig and dependency graph read. The lockfile covers dependency versions, and the pins inside the package cover the Rust toolchain. An input missed here is a bug to fix by adding it to the set; the fallback direction is always a rebuild, never a skip. - The whole `patches` directory is an input, although the Runner build reads three of its fourteen files. An edit to an unrelated patch therefore costs one extra Runner rebuild, which is the safe direction and avoids a hand-kept list of file names that could drift from the scripts that read them. - The stamp lives in `node_modules/.cache`, which pnpm does not manage and `.gitignore` and `.dockerignore` already exclude. It is about 128 KB with the output manifest. The Docker build has no stamp and no prior Runner build, so it rebuilds exactly as before. - A stamp is written only after the Runner's `build` passed every step on inputs that did not change while it ran. The Runner's first step and the prepare step both remove any existing stamp before a rebuild starts, so a build that stops partway, or a build command that fails before its first step, leaves no stamp. - `node scripts/record-build-inputs.mjs --finish` on its own is an error; the script is only meant to run as the two ends of the `build` chain, and the guard test pins that. - The pending record is keyed by the pid of the shell that runs the chain. pnpm runs every script through a shell (`sh -c` on POSIX, `cmd` on Windows), and both phases are children of that shell; a shell that execs the last command of the chain is covered by matching the finishing process's own pid too. A chain run some other way, with the two phases under different parents, fails loudly at `--finish` rather than recording. Two builds running at once still write the same `dist`, which no stamp can make safe; what the record guarantees is that a stamp is written only by a build whose own inputs did not move. - The platform and architecture are part of the comparison, so a stamp from another machine is never trusted; an older v1 stamp is rejected by schema. - The Typecheck job builds the Runner once, through this step, with no prior `pnpm -r build`, so it gains nothing and loses nothing: there is no stamp and it rebuilds as before. - The step now spawns pnpm through `npm_execpath` instead of relying on `pnpm` being on PATH, because under corepack the script shell has no `pnpm`. Outside a pnpm script it still falls back to `pnpm` on PATH. ## Model Used - Claude (Anthropic), model id `claude-fable-5-1` (Claude Fable 5.1), extended thinking, with tool use (shell, file editing, GitHub CLI) in a Paperclip agent session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-10-12 11:25:24 -07:00
DottaandPaperclip	ce08872dfb	refactor(heartbeat): extract instruction working-copy lifecycle (#15984) ... ## Thinking Path > - Paperclip runs agent tasks and preserves their managed agent files. > - Heartbeat must prepare a private copy, collect stopped-provider edits, and retain ownership during warm-session handoff. > - That lifecycle is spread across preparation, adapter callbacks, and cleanup. > - A per-run controller can own the copy and receipt while the executor retains teardown ordering. > - This PR moves that coordination into a dedicated module. > - Live getters preserve updated state and partially acquired resources. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat instruction preparation. This is the second and final PR in this batch, originally stacked on #15980. That dependency is now merged; this PR is rebased onto its squash merge with an identical patch and identical touched-file contents. Related-PR search found no duplicate extraction. **Current behavior** After #15980, `heartbeat.ts` has 4,215 lines and 173,657 bytes. It coordinates instruction-copy preparation, save receipts, warm handoffs, and deferred cleanup inline. **Proposed behavior** `heartbeat/instruction-working-copy.ts` owns the per-run lifecycle in 233 lines. `heartbeat.ts` becomes 4,055 lines and 162,520 bytes. The executor retains the outer release order, unavailable-copy reporting, and environment/service cleanup policy. **Reason and benefit** The copy, save receipt, and warm ownership are readable together. The remaining executor names preparation and dispatch phases without obscuring teardown decisions. **Breaking changes** None intended. All 140 named public heartbeat exports remain unchanged. ## What Changed - Extract copy preparation, save/checkpoint callbacks, warm reservation/adoption, prompt guidance, and the deferred-release closure. - Read the executor's current run row through a callback; expose current copies, receipts, and acquired cleanup handles through live getters. - Keep cleanup handles available after preparation failure, and preserve release-before-retry on invalidated warm reuse. - Add four public-path characterization tests with real disposable PostgreSQL and files, plus four direct boundary tests. - Add isolated browser acceptance with a real process that edits instructions, completes its task, and reads the saved bytes on its next task. - Document the boundary and opt-in acceptance command. ## Verification - **Before extraction:** four new public-path tests pass on the original implementation, as do 108 existing instruction-copy/directory/native-cleanup tests. The two-task browser journey also passes on the baseline server. - **After extraction:** all 112 baseline tests pass; all four new boundary tests pass; 12 selected native-session ownership/retirement cases pass (the other 670 cases were outside this focused selection). - **Browser acceptance:** the fresh extracted-code server passes the full two-task journey (25.9 seconds): real child-process edits, saved/unchanged receipts, canonical bytes through the agent-file API, next-task restore, completed task UI, and copy cleanup. - `pnpm -r typecheck`, final server typecheck, and `pnpm build` pass. Module-boundary and public-export checks pass. - Normalized syntax comparison confirms all moved bodies and the reconstructed executor match the base, accounting explicitly for live run getters and delegation. - Final rebased head: `a08872c97e587a21ffc0fc6b78f34b81e484e7b0`. Combined entry-point and boundary regressions pass again (12/12); fresh Greptile review is 5/5 with no findings. All 55 current-head checks pass (two configured skips), including the hosted full-suite tests, build, typecheck, and all eight browser shards. There are no open review threads. ## Risks - Acquisition can succeed before preparation throws. Boundary tests verify that both the reservation and a transferred copy remain visible to the cleanup owner. - Callbacks can replace copy/save state during execution. Public tests verify saved receipts; direct tests verify the current run row, warm checkpoint receipt, and permission-root preservation. - Missing edit authorization and cleanup failure must remain nonfatal. Characterization tests revoke actual membership after dispatch admission and inject a cleanup error while preserving the saved result. - The browser uses a test-launcher-only instruction-capable wrapper around the real process adapter. Production adapter capabilities are unchanged. No live provider calls are made; native warm ownership is covered by focused module and native-session tests rather than a live provider browser run. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 10:52:32 -05:00
DottaandPaperclip	94e4218310	refactor(heartbeat): extract runtime service coordination (#15980) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat runs start workspace services and receive service reports from adapters. > - Service context updates and readiness notices are spread through the executor. > - These steps share a clear responsibility, while process and lease ownership already live elsewhere. > - This pull request puts service coordination in one module. > - The executor retains the same dispatch and cleanup order. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This is the first of two final focused extractions after #15973. Related #15653 changes runtime callback origins; this PR preserves that policy. **Current behavior** `heartbeat.ts` has 4,372 lines and 178,459 bytes. It validates service intents, starts services, persists adapter reports and primary URLs, and posts readiness notices inline. **Proposed behavior** `heartbeat/runtime-services.ts` owns those steps in 212 lines. `heartbeat.ts` becomes 4,215 lines and 173,657 bytes. Process registries and leases stay in `workspace-runtime.ts`. The executor still releases services in its original cleanup path. **Reason and benefit** Service coordination has named preparation and result phases. The execution path is easier to read without changing service lifetime or moving cleanup policy. **Breaking changes** None intended. All 140 named public heartbeat exports remain available, including `postWorkspaceReadyComment`. ## What Changed - Move service-intent validation, startup, adapter-result persistence, and readiness notices into one module. - Preserve mutable run context, primary URL selection, notice error handling, and release ordering. - Add four public-path tests using disposable PostgreSQL, real HTTP service processes, and a fixture adapter. - Extend browser recovery acceptance to fetch the actual service URL, check the readiness notice, and verify listener cleanup. - Update the module map and acceptance instructions. ## Verification - Before extraction: all five tests pass, including the four new characterization cases and existing notice formatting test. - After extraction: all 17 tests pass across service coordination, notice formatting, output recording, and runtime authorization. - Structural comparison confirms that all moved bodies and the reconstructed executor match the baseline. Module boundary checks pass. - `pnpm -r typecheck` and `pnpm build` pass. - Extended isolated browser acceptance: **2/2 pass** (3.7 minutes), covering actual service readiness, recovery, bounded failure, and cleanup. At `8e33e2032de681082f128470eb8ab0711b382afe`, all 55 current-head checks pass (two configured skips), including hosted full-suite tests. Greptile is 5/5 with no findings or review threads. ## Risks - Service context must be persisted before adapter dispatch and before run completion. The tests inspect both phases and preserve the first service URL when adapter reports arrive. - Notice failures must remain nonfatal. The public-path test checks both startup and result notices. - Shared services must survive run cleanup; ephemeral services must stop after success or failure. Tests use actual processes and HTTP requests. - The fixture supplies UUID service report IDs, as required by the database. The existing generated-ID fallback for reports without IDs is outside this extraction. - Test adapters and services are local fixtures; these tests make no live provider calls. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 10:40:42 -05:00
DottaandPaperclip	914b70572c	refactor(heartbeat): extract execution configuration (#15973) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat turns task, agent, environment, and account settings into an executable run. > - The executor still assembles scoped credentials and skill configuration inline between environment selection and session preparation. > - That phase has a clear input/output boundary, but credential acquisition must remain visible to cleanup even when later setup fails. > - This pull request extracts configuration assembly and immediately notifies the executor of acquired resources and redaction state. > - The executor retains admission, dispatch, and cleanup, with the same statement ordering and explicit busy-account deferral. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This PR follows #15969 and #15966, both now merged. The branch is rebased onto `master`; `git range-diff` confirms that the extraction patch is unchanged. **Current behavior** `heartbeat.ts` has 4,561 lines and 188,950 bytes. Configuration assembly mixes workspace overrides, secrets, managed AI preparation, and skill delivery into the executor. **Proposed behavior** `execution-configuration.ts` owns this preparation phase in 269 lines and 16,204 bytes. The executor calls it between environment and session preparation and retains immediate cleanup ownership. `heartbeat.ts` becomes 4,372 lines and 178,459 bytes. **Reason and benefit** The execution path exposes another named phase, with tests that exercise its effective configuration, resource lifetime, failure diagnostics, and busy-account result through public heartbeat dispatch. Related work: #11827 changes pre-dispatch secret refresh and #15355 changes GitHub delegation; this extraction preserves current policies. **Breaking changes** None intended. Public exports remain available, and the original helpers keep their modules. ## What Changed - Move configuration merging, scoped secret resolution, managed AI acquisition, runtime skills, connector instructions, and managed identity preparation into one typed module. - Notify the executor immediately when failure-redaction values, managed credentials, and identity redaction become available. Returning resources only after all preparation succeeds would lose cleanup on partial failure. - Preserve the live failure-secret getter for later runtime environment changes. Return an explicit deferred result for busy accounts. - Add four real-database characterization tests through `heartbeat.wakeup`, using actual credential files and cleanup with a fixture adapter. - Update the developer module map. ## Verification - Before extraction: all 45 tests pass across the new public-path tests and existing project-environment tests. - After extraction: 47 tests pass across execution configuration, project environment, and runtime skills. No skips. - Two additional real-process managed identity tests pass; nine unrelated cases are deliberately filtered out. - `pnpm -r typecheck` and `pnpm build` pass. - Normalized syntax comparison confirms the moved body and reconstructed executor match after accounting for immediate ownership callbacks and the explicit prepared/deferred result. - Isolated browser acceptance: 2/2 pass (3.9 minutes). A real process worker completes after Git recovery, with persisted stdout and matching final output byte counts. Persistent failures stop after three attempts. Both outcomes survive reload. The disposable server was shut down after verification. - All eleven new public-path characterization cases across the three-PR stack pass together on the final commit. - All 140 named public exports from the batch baseline remain available. - CI initially failed the legacy-continuation message browser case: periodic recovery escalated the issue, then queue admission cancelled the run with `execution_reconciliation_required` before configuration preparation. The unchanged predecessor shard passes, and the exact case passes locally (5 seconds in the browser). The original failing case passed on its single rerun (8.2 seconds), but GitHub then shut down the runner during a different test (`The runner has received a shutdown signal`). The separate infrastructure restart completed: 24 browser cases passed, with two existing optional cases skipped. Assertions and timeouts are unchanged. - Greptile: 5/5 on `27ab7e3df64c157ba1f00c5757db812ff7a4e5aa`, with no actionable findings. Hosted full-suite CI passes on this exact commit: all 54 check runs and the additional commit status are successful. - Merge preparation: all eleven characterization tests pass after rebasing onto `master`. Fresh full CI passes for `1758673f87eaa76a8b2c520eac4b631ada5d9ba1`: 55 successful checks and two optional Storybook skips, including all eight browser shards, typecheck, tests, and build. Greptile is 5/5 on this exact commit with zero review threads. All heartbeat source, tests, documentation, and browser assertions are identical to the previously reviewed final stack. ## Risks - A failure after AI acquisition must still remove the disposable credential home. The new public-path failure test proves actual filesystem cleanup and diagnostic redaction before the external reporting sink. - The baseline exposed an existing gap: arbitrary exception text in the local `run.error` field can contain a credential even though the failure reporter redacts it. This behavior is unchanged and merits a separate fix; the test does not claim the local error field is sanitized. - Account-busy deferral must return without dispatch while retaining its scheduled retry. The new test checks the durable run states. - Browser acceptance uses a real process fixture and disposable data, without live provider calls. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 10:01:00 -05:00
DottaandPaperclip	37913e4907	refactor(heartbeat): extract run output recording (#15969) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat runs stream adapter output and persist execution results. > - Output recording currently sits inside the executor beside admission and cleanup. > - Log handles, sequence numbers, progress writes, and excerpt buffers share one attempt lifetime. > - This pull request moves that state and its callbacks into a per-attempt recorder. > - The executor keeps dispatch and finalization visible, while finalization reads live output values. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This PR follows #15966, which is now merged. The branch is rebased onto `master`; `git range-diff` confirms that its extraction patch is unchanged. **Current behavior** `heartbeat.ts` has 4,740 lines and 195,258 bytes. It directly owns log handles, excerpt buffers, output sequence numbers, progress flushes, and adapter event callbacks. **Proposed behavior** `run-output.ts` owns the per-attempt recorder. The main file has 4,561 lines and 188,950 bytes; the recorder has 209 lines and 7,757 bytes. **Reason and benefit** Output state has one explicit lifetime and interface. The shared log store and finalization policy retain their original owners. Related open output work includes #12539 and #15524; this PR does not change redaction policy. **Breaking changes** None intended. Public flush constants and log formatting exports remain available from `heartbeat.ts`. ## What Changed - Move log initialization, redaction, excerpts, progress persistence, live delivery, and adapter callbacks into `createHeartbeatRunOutput`. - Preserve the acquired log handle if later setup fails. Bind callbacks to the execution-start run row. - Pass live excerpt getters to success and failure completion so output arriving during finalization remains visible. - Add three real-database public-path tests. Extend browser acceptance to inspect persisted worker stdout and output byte counts. - Update the module map. ## Verification - Before extraction: 23 tests pass, including all three new output characterization cases. - After extraction: 45 tests pass across output, log formatting, event sequencing, and run completion. No skips. - `pnpm -r typecheck` and `pnpm build` pass. - Normalized syntax comparison confirms the moved bodies and reconstructed executor match. Reviewed the factory ownership, callback binding, getter wiring, and constant re-exports separately. - Isolated browser acceptance: 2/2 pass, including persisted worker stdout and matching final byte counts. The process worker completes after recovery; persistent timeouts stop after three attempts. Both terminal states survive reload. - Hosted full-suite CI passes on `d32c8fc40e72b9766bdd9e191d777e8dc554e3f2`; all 55 reported checks are successful. Greptile is 5/5 with zero review threads. - The initial chat shard timed out waiting for GitHub mention admission. Its fixture mocks heartbeat wakeup and never calls the output recorder. Both parameterized cases passed locally; the unchanged shard passed on one rerun. No test or timeout changes. - Merge preparation: all seven session-preparation and run-output regression tests pass after rebasing onto `master`. Fresh full CI passes for `60e2a49ce2879645a9f618a801e0ba0903ede161`: 55 successful checks and two optional Storybook skips, including all eight browser shards, typecheck, tests, and build. Greptile is 5/5 on this exact commit with zero review threads. ## Risks - A stale excerpt snapshot could lose output that arrives while finalization awaits I/O. Passing-before/passing-after tests exercise this for both successful and failed runs. - Per-attempt state must not become service-wide. Each executor constructs its own recorder and keeps the existing shared store. - Browser acceptance uses a real process fixture and disposable data; it makes no live provider calls. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 09:51:17 -05:00
DottaandPaperclip	848e659bc6	refactor(heartbeat): extract session preparation (#15966) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution prepares a provider session before dispatch. > - The same executor also owns workspace setup, admission, cleanup, and recovery. > - Session freshness and compaction form a separate responsibility with explicit inputs and results. > - This pull request moves those two session phases into one module and keeps their order around workspace preparation. > - The benefit is a shorter executor with unchanged session behavior. ## Linked Issues or Issue Description **What existing behavior does this improve?** Maintenance of heartbeat execution. This continues the extraction series after #15872. **Current behavior** `heartbeat.ts` has 4,989 lines and 206,650 bytes. Session config fingerprints, resume selection, credential compatibility, and compaction are embedded in the executor. **Proposed behavior** The executor calls `prepareHeartbeatSession` before workspace preparation and `resolveHeartbeatAdapterSession` after it. `heartbeat.ts` has 4,740 lines and 195,258 bytes. The new cohesive module has 385 lines and 19,518 bytes. **Reason and benefit** A reviewer can find session policy in one module. Runtime acquisition, admission gates, and cleanup remain in the executor. Related open work includes #15787 and #15142; this PR does not change either policy. **Breaking changes** None intended. Existing public helper exports remain available. ## What Changed - Move 267 lines into `server/src/services/heartbeat/session-preparation.ts` with typed task, config, workspace, and session inputs. - Keep config fingerprints and context mutations on the same objects. Return compaction and adapter-session results to existing consumers. - Add four real-PostgreSQL characterization tests through heartbeat wakeup and adapter dispatch. - Document the two session preparation phases. ## Verification - Before extraction: four new characterization tests pass; 180 existing workspace/session tests pass. - After extraction: 205 tests pass across session preparation, workspace/session policy, managed-account session compatibility, and accepted-plan refresh. No skips. - `pnpm -r typecheck` and `pnpm build` pass. - Normalized syntax comparison: both moved blocks and the reconstructed executor, including public exports, match the base. - Isolated browser acceptance: 2/2 pass. A real process worker completes after automatic recovery; persistent timeouts stop after three attempts. Both terminal states survive reload. - All hosted CI gates pass on `d33f8506a7`, including the complete test shards, typecheck, build, and browser suites. The duplicate local `pnpm test:run` was interrupted after hosted CI passed; it is not claimed as a full local pass. - Greptile: 5/5 on this head, with no review threads. ## Risks - Session freshness can alter provider context if a dependency is omitted. Typed inputs and passing-before/passing-after dispatch tests cover unchanged resumes, model changes, tool refresh, and compaction. - The module acquires no runtime resource. Workspace preparation still occurs between the two phases. - Live external providers are not exercised; browser acceptance uses a real process fixture in a disposable app. ## Model Used OpenAI Codex, based on GPT-6, assisted with reasoning, code edits, tool execution, and verification. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 09:40:09 -05:00
DottaandPaperclip	ce15fe7c73	refactor(heartbeat): extract task context preparation (#15872) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution prepares task context before it selects an environment and starts an adapter. > - This preparation still occupied 319 lines inside the executor after the earlier extractions. > - It combines continuation summaries, comments, prompt variants, communication guidance, and redaction. > - This pull request moves that responsibility into one typed module under `services/heartbeat/`. > - The executor keeps admission, resource ownership, dispatch, and cleanup ordering together. > - The result makes the preparation phase easier to find and test without changing execution behavior. ## Linked Issues or Issue Description Refs #15825, #15807. This continues the incremental heartbeat extraction series. A search found no duplicate task-context extraction. **What existing behavior does this improve?** Maintenance and review of heartbeat task context preparation. **Current behavior** The executor assembles prompts and trust-filtered context inline. This hides its execution phases among preparation details. **Proposed behavior** Call `prepareHeartbeatTaskContext` at the same point in execution. Preserve statement order, prompt contents, public exports, and the shared run context object. ## What Changed - Move task context assembly to `server/src/services/heartbeat/task-context.ts`. Inputs group task state, policy, and two existing service methods. Results preserve the continuation and wake values used by later runtime selection and dispatch. - Keep company-scoped loading, quarantined-content filtering, secret redaction, and revision hashing in their original order. Add no execution resource acquisition. - Add four public-path integration tests before the move. Cover prompt variants and redacted revisions, quarantined summaries/comments, a mixed-company wake batch, and stale caller fields. - Extend the real process worker in the opt-in browser suite to check persisted prompt variants and the description revision before it completes its task. - Update the module map and acceptance-test documentation. - Reduce `heartbeat.ts` from 5,314 lines / 220,784 bytes to 4,989 lines / 206,650 bytes. The new module has 376 lines / 15,912 bytes. ## Verification - Before the move: 11 existing prompt tests and all four new characterization tests passed against the inline implementation. - Normalized syntax comparison passed for all 319 moved lines and the remaining orchestrator, including public exports. Imports and the typed boundary were reviewed separately. - `pnpm -r typecheck` passed. - `pnpm build` passed. - The clean post-move focused run passed all 26 tests in five files, with no skips: task-context characterization, task-markdown, Slack invitation, accepted-plan workspace refresh, and run-preparation. - The local `pnpm test:run` attempt was interrupted after database failures while the Mac exhausted its shared-memory slots. All 15 affected native warm-ownership database cases passed in an isolated rerun after the browser instance stopped; the other 666 cases in that file were excluded by the test-name filter. The interrupted command is not a full-suite pass. Full CI passed on `6bf86cfdc07e7fe34015f7b719218639f4232510`: 55 successful checks and two optional Storybook checks skipped. Greptile reviewed the latest commit at 5/5; its one earlier worker-test finding is fixed and resolved. - Browser E2E passed both scenarios on the final code: persistent timeouts stop after three runs and survive reload; one timeout recovers without manual Retry, verifies task-context prompt variants/revision and preserved workspace files in a real process, then completes through the run-scoped API. The first recovery attempt exposed an incorrect new test assertion because the fixture title equals its description. The worker now checks the description section specifically; the corrected recovery scenario passed in 55.9 seconds. The original failure trace was preserved. These tests use a deterministic process adapter and make no live model calls. ## Risks This is a behavior-preserving extraction. The main risks are changed closure bindings, missing return values, or changed preparation order. Typed inputs, public-path tests, and syntax comparison cover those risks. The run context is still the same mutable object. The helper acquires no leases or processes. It introduces no schema, API, or prompt policy changes. The first post-move database run skipped cases due to host resource exhaustion. All 26 focused cases passed on the subsequent clean run. Browser E2E passed after the worker assertion fix. The full local suite remains incomplete. All 15 native warm-ownership database cases passed on their isolated rerun; The full CI test suite passed on the final commit. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code execution, and test tools. The host does not expose the exact deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 08:15:21 -05:00
DottaandPaperclip	db2f00947b	fix(runner): include Claude ACP in the server package (#15912) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server ships a copy of Paperclip Runner for native agent execution. > - The Claude ACP bridge is a runner dependency, but it was absent from the server package. > - The runner resolves this bridge dynamically, so the import-based packaging check missed it. > - A Claude session then failed before startup and showed a generic connection error. > - This pull request ships the qualified bridge and reports a missing runtime with repair steps. ## Linked Issues or Issue Description **What happened?** A server install with the vendored runner could not load `@agentclientprotocol/claude-agent-acp/package.json`. The runtime test suggested checking the model or provider connection. The task reported an interrupted runner session. **Expected behavior** The server package includes its qualified Claude ACP runtime. A missing bundled runtime reports an installation error and explains how to repair it. **Steps to reproduce** 1. Use a server install with the vendored runner and no Claude ACP bridge. 2. Select the Paperclip Runner ACP harness and the Claude provider. 3. Run the runtime test or open a session. 4. Observe a missing-module failure before Claude starts. **Paperclip version or commit** Reproduced on `e733bddc45`. This branch is based on current master. **Deployment mode** Self-hosted server built from source. Related: #15224 upgrades the Claude bridge. This PR fixes its omission from the server package and keeps the current qualified version. A future bridge upgrade must update the server pin too. ## What Changed - Add Claude ACP `0.73.0` to the server dependencies and bundled dependencies. - Pin the qualified Claude SDK and bridge dependencies in the npm bundle producer graph. Publish its native packages as consumer optional dependencies instead of shipping the producer executable. - Bind source-built vendored runners to the shared pnpm dependency store while retaining the server manifest as the provider authority. Keep published npm and self-rooted deployment roots unchanged. - Include dynamically resolved Claude and Codex bridges in the vendor dependency check. - Preserve the missing-package category through the sidecar, Rust transport, and native session executor. Stop automatic retries for this installation failure. - Show **Runtime missing** in the runtime test and task thread. Explain the reinstall or rebuild, server restart, and runtime test steps. - Add regression tests and document the repair procedure. ## Verification - Red/green comparison with the production sidecar: the same missing-package setup returns a raw require-stack error on the original code and an actionable, non-retryable missing-runtime error on this branch. - The rebuilt server's vendored Claude installation probe passes using the source-server authority selected by the launcher, without starting a provider or making a model request. - Focused server verification: 830 tests passed. Runner installation and sidecar error tests: 83 passed, with 6 platform-specific skips. Packaging tests: 28 passed. Rust transport tests: 7 passed. - Source/deployment authority tests: 3 passed. Red/green packaging tests include a Linux-produced Claude SDK bundle, actual npm pack, and closed executable admission on Linux x64 and macOS ARM64/x64. - Full UI suite: 8,113 tests passed. - `pnpm -r typecheck` and `pnpm build` passed on the rebased branch. - Full current-head CI passed, including all general and serialized test lanes, Runner Rust/Vitest verification, typecheck, build, and all eight e2e shards. Greptile reviewed `d6a2741834ff922956560f2f689382827bbaf9e0` at 5/5 with no unresolved threads. - Local full-suite limitation: an earlier parallel attempt hit Postgres startup failures and timeouts; the affected server tests and a database fixture passed in isolation. The subsequent serial `pnpm test:run` was stopped while still running after complete CI passed. Focused local tests, full UI tests, typecheck, and build completed as listed above. - Token gates and `git diff --check` passed. ## Risks - Bundling the Claude bridge increases the server package size. The producer graph retains the qualified SDK version and existing isolation patch. Native platform declarations move to the published server manifest, while the runtime keeps its exact version, declared-slot, no-follow, and executable digest checks. - The new failure category is permanent. Operators must repair the server installation before they retry the task. - Existing failed runs retain their recorded error. The clearer message applies to new runtime tests and runs. - No database schema or migration changes. ## Model Used OpenAI GPT-6 through Codex. The system reports the model family; the exact deployment ID and context-window size are not exposed in this session. Used reasoning, browser inspection, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-12 06:51:22 -05:00
4 people	85f4205de3	perf(test): reuse a migrated embedded Postgres template across suites (#15861) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pull requests to this repository run a large CI matrix. The slowest check sets the time a contributor waits for a green run. > - After #15853, the twelve `server (n/12)` shards are the slowest check in every fully green PR run created since its merge (518s to 548s; `Build` is 482s to 504s). Each shard is about 60s of setup and 454s to 490s of Vitest, with one worker, so its wall time is the sum of its suites. > - From two shard logs, 22 to 24 of the 71 to 74 suites in a shard start an embedded Postgres cluster. Each one runs `initdb`, starts the cluster, and applies all 331 migrations before its first test. The smallest such suite takes 6s with one or two trivial tests. On top of that, the `getEmbeddedPostgresTestSupport` probe starts and stops a second throwaway cluster at import time in 19 of those suites, which is why their per-file import overhead is 3.5s to 4.1s against 0.9s to 1.0s for suites without Postgres. > - A cluster data directory that was shut down cleanly can be copied and started again as an independent cluster, and the migrated schema is the same for every suite. > - This pull request builds one migrated cluster per host, publishes its data directory as a template under the OS temp directory, and starts every later suite from a copy of it. The probe reuses the template instead of starting a cluster. > - The benefit is about 7s to 8s per Postgres suite on the CI runners (measured 3.2s to 0.35s per start and 0.75s to 0.03s per probe locally), about 120s to 170s per server shard, and a faster local test loop for every suite that uses the fixture. ## Linked Issues or Issue Description **What existing behavior does this improve?** The `server (n/12)` PR checks are the slowest check in every fully green PR run since 2026-10-11 05:27Z. Their Vitest step runs suites one at a time, and about a third of each shard's time is embedded Postgres setup that produces the same migrated cluster every time. **Subsystem affected** Test infrastructure: the embedded Postgres test fixture in `@paperclipai/db` (`packages/db/src/test-embedded-postgres.ts`), used by 349 server suites, the db package suites, and the CLI suites. **Current behavior** `startEmbeddedPostgresTestDatabase` runs `initdb`, starts the cluster, creates the database, and applies all 331 migrations for every suite. `getEmbeddedPostgresTestSupport` starts and stops a second throwaway cluster once per test file, because the module-level cache does not survive Vitest's per-file isolation. Measured locally: 3.2s per suite start plus 0.4s cleanup, and 0.75s per probe. On the CI runners the smallest Postgres suite takes 6s and the probe adds 2.5s to 3s of import time per file. **Proposed behavior** The first caller on a host builds one migrated cluster, stops it, and renames its data directory to `$TMPDIR/paperclip-embedded-postgres-templates-<uid>/<key>`. Every later caller copies the template (about 90 MB) and starts Postgres on the copy, then still runs `ensurePostgresDatabase` and `applyPendingMigrations`, which are no-ops on a current template. The probe treats a published template as proof that the host can run the cluster. The key hashes the embedded-postgres version, the initdb flags, the user, and the name and content of every migration file plus the journal, so a changed migration builds a new template. `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` bypasses the cache. Any template failure falls back to the fresh path. A template is a Postgres data directory, and starting it executes its configuration, so the cache is private to the current OS user: the root name carries the uid, the root is created with mode 0700, and both the root and each template are used only when `lstat` shows a real directory (not a symlink) owned by the current user with no group or world write bit. A root that fails those checks (for example one pre-created by another user in a shared `/tmp`) is left alone and the process uses fresh clusters. **Reason and benefit** The slowest check decides the wall-clock time of every PR run. Measured locally, a template start takes 0.3s to 0.4s instead of 3.2s, cleanup 0.15s instead of 0.43s, and the probe 0.03s instead of 0.75s. Four server suites that passed in both modes dropped by 3.4s to 6.0s each; the db `backup-lib` suite dropped from 42.7s to 20.9s. The CI runners are about twice as slow, so the expected saving per server shard is 120s to 170s. **Breaking changes** None. The returned database is fully migrated in both paths, each suite still gets its own cluster and data directory, and a failure on the template path falls back to the existing fresh path. ## What Changed - `packages/db/src/test-embedded-postgres.ts`: adds the template cache. `computeEmbeddedPostgresTemplateKey` hashes the inputs that determine a migrated cluster. `buildEmbeddedPostgresTemplate` starts a fresh cluster through the existing bounded retry, creates the database, applies migrations, waits for the real `stop()` (30s bound), and publishes the data directory with an atomic rename; a concurrent builder that loses the rename discards its copy. `startEmbeddedPostgresFromTemplateWithRetry` copies the template into a fresh temp directory, sets mode 0700, and starts the cluster with the same per-attempt port and directory rotation as the fresh path. `startEmbeddedPostgresTestDatabase` tries the template first and falls back to the fresh path with one warning per process. `probeEmbeddedPostgresSupport` returns supported when a template exists or can be built, and keeps its throwaway-cluster path for the case where the template could not be published. Templates for other keys are pruned after seven days. The connection strings and initdb flags move into shared constants; no other behavior changes. - `packages/db/src/test-embedded-postgres.test.ts`: two unit tests for the key (stable for identical inputs; changes with migration content, a new file, flags, version, or user; hashes this package's real migrations directory) and two real-cluster tests (the published template is complete and private, a suite starts from it in `template` mode with migrations up to date, a table written to one copy is not visible in the next copy, and `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` runs the `fresh` path). The real-cluster tests skip on hosts without embedded Postgres, like every other fixture suite. - `doc/DEVELOPING.md`: a short section under Test Commands describing the template, its key, and the bypass variable. **Review fixes (second commit), one per finding:** - Private cache root (Greptile P1, Superagent P2). The root is `$TMPDIR/paperclip-embedded-postgres-templates-<uid>` (user name on Windows), created with mode 0700. `prepareTemplateRootDir` and `isUsableTemplate` both reject, via `lstat`, anything that is a symlink, not a directory, owned by another uid, or writable by group or others; a rejected root logs one warning and the process uses fresh clusters. A root created by the first commit with a wider mode is narrowed to 0700 if this user owns it. - Template failures never skip suites (Greptile P1). `getEmbeddedPostgresTemplateDir` no longer rejects. A build or root failure resolves `null` with one warning, and `probeEmbeddedPostgresSupport` then runs the throwaway-cluster probe, so only a cluster that cannot start reports `supported: false`. A migration that cannot apply fails the suite on the fresh path instead of being hidden by `describe.skip`. - Idle-based pruning (Greptile P2). Each reuse refreshes the template's timestamp with `utimes`, so the seven-day prune measures idle time. A template that disappears after a process resolved its path is not retried five times: `startEmbeddedPostgresFromTemplateWithRetry` checks the template before each copy and throws at once, the suite falls back to a fresh cluster, and the fallback forgets the path so the next caller in that process rebuilds the template. - The template-path test enables the cache for itself and restores the original `PAPERCLIP_TEST_POSTGRES_TEMPLATE` value afterwards (Greptile P1), so the suite passes under `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` as well. - Eight new tests: `isUsableTemplate` accepts a private cluster directory and rejects a missing `PG_VERSION`, a leftover `postmaster.pid`, a symlink, and (POSIX) group- or world-writable modes; a world-writable pre-created root and a symlinked root both fall back to fresh clusters, publish nothing, log one warning, and still report `supported: true` from the probe; a ten-day-old template is refreshed on reuse; a pruned template is rebuilt on the next start; and a template build that fails after the cluster started falls through to the fresh probe. The tests that need an empty cache point `os.tmpdir()` at a scratch directory (`TMPDIR`, `TEMP` and `TMP` together, so the redirect holds on Windows as well) and restore it. **Review fixes (third commit, e8f8e9c8), one per finding:** - Fallback test on Windows (Greptile P1). `useScratchTmpdir` set only `TMPDIR`, which Windows `os.tmpdir()` ignores, so the one scratch-root test that runs on every platform would assert on paths outside the scratch root there. The helper now sets and restores `TMPDIR`, `TEMP` and `TMP` together, and that test asserts `os.tmpdir() === scratch` before it checks any path under it. - Timestamp test runs on its own (Greptile P2). The test assumed an earlier test had published a template; under `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` the probe does not, so a name-filtered run threw `ENOENT` from `utimes`. It now starts and cleans up one suite first, which builds the template when nothing has, and asserts `isUsableTemplate` before ageing it. ## Verification - `pnpm --filter @paperclipai/db exec tsc --noEmit`: clean (both commits). - `pnpm --filter @paperclipai/db exec vitest run src/test-embedded-postgres.test.ts`: 15 pass (12 new across both commits) with the template root absent (first run builds it), with it present, and with `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0`. - After the review fixes: `client.test.ts` and `backup-lib.test.ts` (29 tests) pass on the template path, and `agent-action-audit-routes.test.ts` and `status-cards.test.ts` (36 tests) pass through the server route harness. The second commit's CI run ([38121021162](https://github.com/paperclipai/paperclip/actions/runs/38121021162)) passed every check: `server (n/12)` jobs 277s to 425s, Vitest `Duration` 219s to 365s, no fallback warning in any shard. - Third commit: reproduced the Greptile P2 on the old file first (`PAPERCLIP_TEST_POSTGRES_TEMPLATE=0 vitest run src/test-embedded-postgres.test.ts -t "refreshes the template"` on an empty cache root fails with `ENOENT … utime`), then with the fix the same command passes, and the full file passes 15/15 on a cold cache, a warm cache, and with the bypass set. `tsc --noEmit` clean. **Measured on this PR's first CI run ([38117892083](https://github.com/paperclipai/paperclip/actions/runs/38117892083)), which ran the branch's fixture in every server shard:** | | Before (runs 38114982393, 38115279131, 38115459196) | This PR's run | | --- | --- | --- | | Vitest `Duration` per `server (n/12)` shard | 454s to 490s | 224s to 337s | | `server (n/12)` job | 518s to 548s | 286s to 401s | | Slowest check in the run | a server shard | `Build`, 453s | No shard logged a fixture fallback warning, so every Postgres suite in the run started from the template. - `pnpm --filter @paperclipai/db exec vitest run src/status-card-migrations.test.ts src/nested-skill-folders-migration.test.ts src/chat-github-migration.test.ts src/question-response-delivery-migration.test.ts src/client.test.ts`: 5 files, 24 tests pass on the template path. These suites drop indexes, re-run migration files, and create extra databases on the cluster, so they cover the semantics the template must preserve. - Four server suites run with `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` and then with the template, all passing in both modes: | Suite | Fresh path | Template path | | --- | --- | --- | | `server/src/__tests__/agent-action-audit-routes.test.ts` (11 tests) | 17.9s | 14.6s | | `server/src/__tests__/status-cards.test.ts` (25 tests) | 22.2s | 18.6s | | `server/src/__tests__/plugin-managed-routines.test.ts` (3 tests) | 18.2s | 14.3s | | `server/src/__tests__/issue-thread-interactions-service.test.ts` (106 tests) | 19.0s | 12.9s | | `packages/db/src/backup-lib.test.ts` (9 tests) | 42.7s | 20.9s | - Per-operation timings on the same host: fresh `initdb` + start + create database + 331 migrations 3.2s, cleanup 0.43s; template copy 0.20s to 0.23s, start from copy 0.03s to 0.04s, `ensurePostgresDatabase` 0.013s, `applyPendingMigrations` fast path 0.03s to 0.04s, stop 0.03s. Probe with the template on disk 0.03s (was 0.75s). Template size 87 MB, 3215 files. - This PR's own server shards run the branch's fixture, so their Vitest `Duration` lines show the saving directly. Expected on this PR's run: `server (n/12)` about 330s to 430s (was 440s to 548s on runs 38114982393, 38115279131 and 38115459196), and the slowest check moves to `Build` (482s to 504s). **Measured on the three fully green `master` PR runs created after #15853 merged** (slowest server shard logs): | Run | Server shards | Vitest `Duration` (slowest shard) | Postgres suites in it | Their reported duration | Their import overhead | | --- | --- | --- | --- | --- | --- | | 38115459196 | 476s to 548s | 483.8s (tests 72%, import 24%) | 22 of 71 | 197s | 83s | | 38114982393 | 409s to 518s | 452.8s (tests 65%, import 30%) | 24 of 74 | 264s | 98s | ## Risks - Low risk to correctness. The template is a cleanly stopped cluster produced by the same `initdb` flags and the same migration runner as the fresh path, and each suite still runs `ensurePostgresDatabase` and `applyPendingMigrations` on its copy. Each suite gets its own copy, its own port, and its own data directory, so isolation between suites is unchanged. - A stale template is prevented by the key. If a migration file, the journal, the embedded-postgres version, the initdb flags, or the user changes, the key changes and a new template is built. Even if a stale template were used, `applyPendingMigrations` on the copy would bring it up to date. - Concurrent first use (for example the file-parallel db project) can build the same template twice. The rename is atomic, so one copy wins and the other is discarded; a template that exists is always complete. The loser pays one extra fresh start, which is what every suite paid before. - The template root lives under the OS temp directory and holds about 90 MB per key. Keys that have not been used for seven days are pruned when a new template is built; each reuse refreshes the timestamp. - A template is executable configuration, so the cache is only used when the root and the template are private directories of the current user (ownership, no symlink, no group or world write bit). On a host where another user pre-created the path, the process logs one warning and uses fresh clusters. Windows has no uid, so there the check is directory-only; its temp directory is already per user. - Any failure on the template path (root checks, build, copy, start, publish) falls back to the fresh path and logs one warning per process, so a broken cache can make a suite slower but never fail it or skip it. The guard tests assert the mode, so a silent fallback on CI would fail `test-embedded-postgres.test.ts` rather than hide. - Hosts that cannot run embedded Postgres still get `supported: false` with the real Postgres output, now from the throwaway probe: a failed template build is one warning, then the probe decides. On such a host the probe costs one extra failed start attempt cycle per file compared with the first commit; every suite on that host skips anyway. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, running as a Paperclip agent through the Claude local adapter with tool use (shell, file edits, GitHub CLI). The agent collected the CI data, made the change, ran the tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (53 checks green on both run 38117892083 and run 38121021162; the third commit re-runs them) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (1/5 on the first commit, 4/5 on the second; every finding so far is fixed and answered, and Superagent confirmed its fix; re-review of the third commit pending) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Bender (Fable) <bender-fable@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>	2026-10-11 11:19:13 -07:00
DottaandPaperclip	30e338fba6	refactor(heartbeat): extract environment selection (#15825) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution selects an environment before it prepares a workspace and starts an agent. > - Workspace preparation now has its own module, but its selection policy still occupies the executor. > - Environment defaults, recovery bindings, chat isolation, and workspace contention must retain their order. > - This pull request moves that selection phase behind one typed function. > - The executor keeps dispatch gates and lease cleanup. Tests cover the public path and the new boundary. ## Linked Issues or Issue Description Refs #15807. This continues the heartbeat extraction series after workspace preparation. **Current behavior** The 5,599-line heartbeat service contains a 301-line environment selection phase inside run execution. **Proposed behavior** Move that phase to `heartbeat/environment-selection.ts`. Keep the same policy, statement order, errors, and public exports. **Reason and benefit** Readers can find selection policy in one module and follow the executor's phases without reading each policy branch. ## What Changed - Add `selectHeartbeatEnvironment` with typed run, task, policy, configuration, and service inputs. Return the values needed by configuration and workspace preparation. - Preserve native recovery bindings, environment precedence, managed sandbox enforcement, Kubernetes provisioning, native chat isolation, and shared workspace contention. - Reduce `heartbeat.ts` from 5,599 lines / 234,599 bytes to 5,314 lines / 220,784 bytes. The new module is 402 lines / 18,215 bytes. - Add six public-path characterization cases before moving production code and six boundary cases. Repair plugin-test teardown to drain active execution and call the database cleanup helper. - Assert the selected local environment in the browser fixture's real worker and document the new boundary. ## Verification - Before the move: 71 tests passed, including all six new public-path cases. Three plugin tests skipped when the host could not start their database probe. - After the move: all 112 focused tests passed across eight files, with no skips. These cover environment policy, local and plugin execution, contention, recovery bindings, workspace preparation, chat isolation, and low-trust directories. - Passed `pnpm -r typecheck` and `pnpm build`. - A normalized TypeScript AST comparison confirms the 301 moved lines and remaining executor statements are unchanged, apart from the new call and imports. - Browser acceptance: the task UI recovered automatically and the worker verified its selected local environment, preserved uncommitted files, and completed the task. The persistent-failure case passed in isolation: three total runs, then a blocked task that remained blocked after reload. - The first persistent-failure browser attempt missed the retry banner within five seconds, despite a persisted scheduled retry. Its unchanged isolated rerun passed. The trace is retained; the intermittent UI failure is not yet attributed to a root cause. No assertion or timeout was relaxed. - Greptile reviewed `20953dd164decb565bc8d85081aad31a23fd33e5`: 5/5, no actionable findings. - Hosted CI passed on `20953dd164decb565bc8d85081aad31a23fd33e5`: 52 successful checks and two intentionally skipped Storybook checks, with no pending or failing checks. This includes all general and serialized test groups, Runner verification, eight browser shards, typecheck, and build. - The duplicate local `pnpm test:run` was interrupted after hosted CI completed. It is not reported as a local full-suite pass. Local completed checks are the 112 focused tests, typecheck, build, and browser cases described above. ## Risks - This selection phase enforces execution restrictions. A wrong input or import could select the wrong environment or change retry behavior. Public-path tests cover refusal, redirect, contention, and dispatch; boundary tests cover recovery and isolation. - The helper can provision a managed environment row and add concurrency notes to the supplied task context. It acquires no execution lease. The executor retains its catch/finally boundaries and resource ownership. - Kubernetes selection is tested with real database state and bootstrap settings. These checks do not claim execution against a live Kubernetes cluster or hosted sandbox provider. - No schema, migration, API contract, or intended behavior change. ## Model Used OpenAI Codex (GPT-6), with repository inspection, code editing, compiler checks, database integration tests, and browser automation. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-11 06:04:31 -05:00
DottaandPaperclip	07fee85eaf	refactor(heartbeat): extract workspace preparation (#15807) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service prepares the workspace and dispatches each agent run. > - After the runtime extraction, `heartbeat.ts` still had 6,417 lines. > - Workspace setup occupied one contiguous 857-line block inside `executeRun`. > - This pull request moves that block into one workspace preparation module. > - The executor keeps dispatch gates and terminal cleanup. Reviewers can inspect workspace setup as one phase. ## Linked Issues or Issue Description Refs #15803 and #15578. This continues the heartbeat refactor series. The search found no open duplicate workspace preparation extraction. **What existing behavior does this improve?** It makes heartbeat workspace preparation easier to maintain. It does not change the workspace policy or execution behavior. **Current behavior** `executeRun` contains workspace resolution, freshness decisions, provisioning, persistence, issue binding, environment acquisition, and realization inline. **Proposed behavior** `prepareHeartbeatWorkspace` owns that sequence in `server/src/services/heartbeat/workspace-preparation.ts`. `heartbeat.ts` calls it at the same point and uses its returned workspace, lease, target, and metadata. **Reason and benefit** The main file falls from 6,417 to 5,597 lines. The workspace phase has a typed boundary and direct regression tests. **Breaking changes** None. There are no schema, API, feature-flag, or execution-policy changes. ## What Changed - Extract the 857-line workspace preparation block without changing its operations or their order. - Preserve native recovery bindings, workspace persistence failure cleanup, issue rebinding after realization, and the cloned restore lease. - Keep environment selection, dispatch admission, terminalization, and final lease release in `heartbeat.ts`. - Add 12 boundary tests with disposable PostgreSQL, real Git repositories, and real local environment leases. - Make the existing Stop and plan-only recovery tests await executor completion instead of short polling deadlines. Assert exactly one continuation for the specific plan-only source run. - Keep server-only Node preload hooks out of the deterministic process adapter used by Stop browser acceptance. - Refresh workspace recovery browser acceptance for the current task composer, and give its disposable test-drive an explicit 10-second snapshot deadline. Production snapshot settings stay unchanged. - Document the workspace preparation boundary in `doc/DEVELOPING.md`. ## Verification - AST comparison: all 857 moved lines match the original block. The remaining module body differs only at the extraction call. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - Focused regression coverage: 651 distinct cases across eight suites passed across targeted runs, including all 12 new boundary tests. The complete 400-test recovery suite passed together on the final commit. - Workspace bootstrap browser acceptance: both journeys passed (4.2 minutes), including preserved source edits, ignored-file exclusion, bounded retries, and persistence across reload. - Stop/queue browser acceptance: both process and native runner journeys passed (2.2 minutes), covering queueing, Stop, subtree pause/cancel, resume, and unrelated-task isolation. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/38062408589): all 47 jobs passed, including the general and serialized test suites, eight browser shards, build, typecheck, native runner checks, and canary dry run. - The local `pnpm test:run` was interrupted to prioritize the recovery regression; the complete repository suite was verified in CI. - Greptile: 5/5 on `c28e698b885cf5fd12f36f7b4052e7a2e6f879d7`, with no open review threads. ## Risks - This code writes durable workspace bindings and acquires environment leases. A changed operation order could break recovery or cleanup after partial setup. The extraction preserves the original statements and exception boundaries. - The new tests cover creation, reuse, immutable recovery bindings, persistence failure, acquisition and realization failure, controller ownership loss, isolation, execution policy, and lease snapshot independence. - Browser acceptance uses real application services with deterministic local workers and a fake Codex protocol provider. It does not qualify live cloud providers. ## Model Used OpenAI GPT-6 through Codex, with code editing, shell execution, and test tools. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-10 10:30:17 -05:00
DottaandPaperclip	1fb3f9746e	refactor(heartbeat): extract prepared runtime execution (#15803) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat prepares, dispatches, and completes each agent run. > - After the runtime-selection extraction, the main service still had 7,295 lines. > - Provider dispatch and workspace finalization formed one large block with shared cleanup rules. > - This pull request moves that block into a typed runtime-execution module. > - The service now has 6,417 lines. Readers can follow admission and outer cleanup without reading the provider execution details. ## Linked Issues or Issue Description Follow-up to #15746. I searched open heartbeat PRs and issues. I found no competing runtime-execution extraction. **What existing behavior does this improve?** The organization and test coverage of native and legacy heartbeat execution. **Current behavior** The heartbeat service contains a 912-line block for provider dispatch, usage capture, workspace finalization, and instruction cleanup. **Proposed behavior** Move that block to `server/src/services/heartbeat/runtime-execution.ts`. Keep the atomic dispatch gate, admission, terminal run status, and outer lease cleanup in the service. Preserve run behavior. **Reason and benefit** This gives runtime execution a named boundary and direct tests. It removes 878 lines from the main service in one bounded change. **Breaking changes** None intended. There are no schema, API, or policy changes. ## What Changed - Add `executeHeartbeatRuntime` for prepared native and legacy execution, usage capture, workspace finalization, and gateway/instruction cleanup. - Return an explicit dispatch result. Update outer ownership state through immediate callbacks, including before exceptions. Read instruction state through live getters. - Move the two internal recovery error classes with their existing behavior. The caller imports the same classes for `instanceof` checks. - Add 15 database-backed execution tests. Keep the existing heartbeat regression suites. - Repair the native Stop acceptance fixture: installed Codex now takes precedence over `PATH`. A test-only loader pins the fake provider. An isolated home and dummy key keep readiness offline. Add four loader tests. - Document the module boundary and fixture setup. ## Verification - Passed: eight focused heartbeat suites (483 distinct tests, including 15 new execution tests). The boundary suite passed again after adding the failed-restore retention case from review. - Passed: `node --test tests/e2e/fixtures/composer-stop-codex-loader.test.mjs` (four tests). - Passed: `pnpm -r typecheck` and `pnpm build`. - Passed: AST comparison of all 912 moved lines after the explicit boundary substitutions. SQL template text, recovery class bodies, public exports, and the rest of the executor match the baseline. - Passed: browser Stop journeys for legacy process and native Paperclip Runner (two tests). The native fixture log proves provider `turn/start` before Stop and `turn/interrupt` after Stop. - Passed: all three ACP continuation browser journeys (same-session queued input, no replay of an interrupted write, and a new direction after composer Stop). Five relevant browser journeys passed in total. - Passed: the full CI test groups, including all server, chat, workspace, serialized-server, runner, and browser shards. - Local full-suite limitation: `pnpm test:run` first hit the macOS shared-memory segment limit while starting embedded PostgreSQL. After releasing only this task’s unused acceptance-server segments, the retry progressed without reported failures. I stopped that duplicate run after the full CI test groups passed. - Passed: all 56 PR checks/statuses on `8694c116e8d43ed8ac11bc4c58623ea3af34a675`. Greptile is 5/5 with no unresolved review threads. ## Risks - Cleanup depends on state written before an error. The module reports dispatch, retention, and recovery ownership immediately. Tests cover early returns and exceptional cleanup. - Instruction callbacks can replace saved metadata during execution. Live getters preserve those changes. - Native acceptance uses a real server, database, browser, and Rust runner with a deterministic provider. It does not verify a live provider service. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and repository/tool access. The exact serving model ID and context-window limit were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-10 09:10:53 -05:00
DottaandPaperclip	2a2071340d	refactor(server): extract heartbeat runtime selection (#15746) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution prepares a runtime, dispatches work, and records the result. > - Earlier refactors separated workspace services, scheduling, and completion. > - The executor still builds native inputs and records runtime selection inline. > - This preparation must preserve admitted inputs and serialize with Stop. > - This pull request moves that phase into a typed runtime-selection module. > - Reviewers can inspect runtime preparation separately from dispatch and cleanup. ## Linked Issues or Issue Description Refs: #15732. **What existing behavior does this improve?** Maintainability of heartbeat runtime preparation and selection. **Subsystem affected** server/ — orchestration services. **Current behavior** The main heartbeat service has 8,128 lines. `executeRun` builds or restores native inputs, selects sessions and sandbox lifecycle, and records native or legacy selection inline. **Proposed behavior** Move this phase into `server/src/services/heartbeat/runtime-selection.ts`. The executor calls `selectHeartbeatRuntime` at the existing boundary and receives an explicit selection result. The main file becomes 7,275 lines, a reduction of 853 lines. **Reason and benefit** This gives runtime selection its own typed boundary and direct tests. It preserves existing decisions, database transactions, and cleanup ordering. **Breaking changes** None intended. All public heartbeat exports remain available. ## What Changed - Move native input assembly, completion contract selection, checkpoint lookup, runtime persistence, and native workspace-sync preparation into `selectHeartbeatRuntime`. - Preserve the row lock and controller lease check that serialize native selection with cancellation. - Report lifecycle and ownership changes to the executor immediately, including when a later preparation step throws. - Move sandbox lifecycle and model fallback helpers with compatible public re-exports. - Add 12 database-backed boundary tests. Update the existing session-recovery source test and developer guide. ## Verification - Structural AST comparison against current `master` confirms the 784-line selection block, moved helpers, and remaining service bodies are unchanged after normalizing boundary callbacks. All 140 public exports are preserved. - On final head `2eddaa485f`, all 12 new database-backed tests and 83 existing lifecycle, selection, fallback-warning, and session regression tests pass (95 total). - Five executor integration cases passed before the rebase. Their final-head rerun could not start because the local checkout was removed during verification; no assertions ran in that last attempt. - `pnpm -r typecheck` and `pnpm build` both pass on final head. - Local verification limitations: the full `pnpm test:run` was attempted, then stopped after database/process startup timeouts in concurrent local runs. Two unchanged optional real-runner session cases timed out on the initial run and one isolated retry; they were excluded from the 95-test focused run. CI also skips those two binary-dependent cases. - All CI gates pass on final head `2eddaa485f`: typecheck, build, general and serialized tests, runner checks, E2E, and canary dry run. - Greptile reviewed final head at 5/5 with no actionable findings. ## Risks This boundary controls native admission and cleanup ownership. Losing an update after a later exception could change resource cleanup. Immediate callbacks preserve those updates. Tests cover Stop races, lost controller leases, immutable admitted inputs, competing runtime selection, and workspace-sync failures. The change adds no schema or API behavior. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code editing, and command execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (95 focused tests on final head; broader local limitations documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-10 07:31:15 -05:00
Dotta	8f13ec4d51	fix(slack): link created tasks in default reply guidance (#15579) ... ## Thinking Path > - Paperclip helps people manage AI agents and their tasks. > - Slack agents use shared guidance to write their replies. > - A reply can show a task ID without a link. > - The person must then search for the task. > - This change tells the agent to link each task it creates. > - People can open the task directly from the reply. ## Linked Issues or Issue Description **What existing behavior does this improve?** Default communication guidance for Slack agents. **Current behavior** The guidance does not require a link when an agent creates a task. **Proposed behavior** Require a clickable link to each created task. Use its identifier or title as the link text. Use a public URL supplied by Paperclip. Report when no public URL is available. **Reason and benefit** People can open the task without a separate search. Related work: https://github.com/paperclipai/paperclip/pull/13760 introduced the guidance. https://github.com/paperclipai/paperclip/pull/15576 adds account connection instructions and does not cover task links. ## What Changed - Add task link instructions to the shared Slack communication guidance. - Document the default and the existing conversation snapshot behavior. - Add regression coverage for the guidance in channels and DMs. ## Verification - Passed: `pnpm exec vitest run server/src/services/chat-communication-guidance.test.ts` (10 tests). - Passed: `git diff --check`. - Local full typecheck and build could not finish: the container killed compiler processes (exit 137; cgroup reports OOM kills). The full test run was stopped after this resource limit. CI results are pending. - Added checks for task links, public URL sources, and missing URLs in Slack channels and DMs. ## Risks - Low risk. This changes prompt text only. - Existing conversations keep their original guidance snapshot. New conversations receive the new default after deployment. - This is agent guidance. It does not force a link into provider output. ## Model Used OpenAI Codex, GPT-6. The runtime did not expose an exact model revision or context window size. Used reasoning, code inspection, file editing, and command execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-09 20:23:18 -05:00
DottaandPaperclip	26d71d8def	Fix human author identity and viewer-relative alignment in task chat (#15494) ... Preserve human author identity in task chat and align only the current viewer on the right. Show other humans and agents on the left with their own identity. Include mixed-author Storybook journeys and regression coverage. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-09 17:04:20 -05:00
DottaandPaperclip	c27ee928b0	refactor(server): extract heartbeat run completion (#15732) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat execution connects provider runs to task state, accounting, and recovery. > - Earlier refactors moved scheduling and lifecycle services into the heartbeat folder. > - The remaining executor still contains result persistence and two large failure handlers. > - These handlers must preserve the outcome chosen by Stop or native recovery. > - This pull request moves them into a typed run-completion module at the same call and catch boundaries. > - Reviewers can now inspect completion separately from provider dispatch and resource cleanup. ## Linked Issues or Issue Description Refs: #15717. Related behavior work: #13857 (agent status convergence). **What existing behavior does this improve?** Maintainability of heartbeat execution and terminal result handling. **Subsystem affected** server/ — orchestration services. **Current behavior** The main heartbeat service has 9,335 lines. Result persistence, accounting, presentation, session updates, and execution/setup failure handling are embedded in `executeRun`. **Proposed behavior** Move those steps into `server/src/services/heartbeat/run-completion.ts`. Keep the existing decisions and ordering. The main service becomes 8,125 lines, a reduction of 1,210 lines. **Reason and benefit** This creates a substantial boundary for the next execution refactor. It also gives terminal handling direct tests without starting a provider. **Breaking changes** None intended. Public heartbeat exports and method signatures remain compatible. ## What Changed - Add `createHeartbeatRunCompletion` with `completeRun`, `failRun`, and `failRunSetup` handlers. - Move terminal-result persistence, accounting, response presentation, retry decisions, and session updates with their failure helpers. - Keep Stop barriers and native ownership/recovery exceptions in the executor. Keep the outer cleanup in its original location. - Pass live output getters and an immediate trace-finalization callback so cleanup sees state even if a later completion step throws. - Add 19 boundary tests, update the existing publication-order source guard to read the extracted module, and document the module in the developer guide. ## Verification - Passed `pnpm -r typecheck` and `pnpm build`. - Structural AST comparison confirms all 74 top-level statements in the three extracted blocks, the moved helper bodies, and all remaining service statements are unchanged after normalizing the explicit boundary parameters. All 140 public exports are preserved. - Passed 304 existing tests across seven adjacent lifecycle, accounting, session, Stop, and presentation suites. After rebase, all 19 new boundary tests and all 12 publication tests pass together. Total focused coverage: 335 passing tests. - Full local `pnpm test:run` was stopped after more than 28 minutes without completing. Additional route-suite diagnosis hit local database startup timeouts. The complete CI run passed on the final head, including all test shards, build, typecheck, browser tests, and canary install checks. - Rebased onto the workspace-repair fix in #15726. Preserve its `defaultBranch` metadata in the moved helper and add a regression test. All 31 completion and publication tests and the server typecheck pass on the rebased head. - Greptile reviewed current head `65e3f8d8ac165dfec4508c467b1b2af8f2510f1a`: 5/5 with no actionable findings. - Final-head [CI run](https://github.com/paperclipai/paperclip/actions/runs/37991844034) passed. All 53 completed check runs succeeded; two optional Storybook checks were skipped. The legacy security status also passed. No unresolved review comments or merge conflicts remain. ## Risks Terminal handling is sensitive to Stop races, late native results, and failures during persistence. The tests exercise these paths with real database status writes, accounting, and task sessions. Native recovery exceptions and resource release stay at their existing boundaries. There are no schema or API changes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code editing, and command execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 16:29:54 -05:00
DottaandPaperclip	65965f0d3a	refactor(heartbeat): extract scheduling and monitor recovery (#15717) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service schedules and runs agent work. > - Previous refactors moved queue admission, recovery, and run control into separate modules. > - Timer ticks, issue monitors, and session-goal recovery still occupy more than 1,100 lines in the main service. > - This pull request moves that scheduling code into one module with explicit dependencies. > - This makes scheduling easier to review and test before we split the execution pipeline. ## Linked Issues or Issue Description Refs #15685, the merged run-control extraction. A search found no duplicate open scheduling extraction. **What existing behavior does this improve?** Maintenance and testing of heartbeat scheduling. **Current behavior** The main heartbeat service contains 10,473 lines. Scheduling shares a file with the execution pipeline. **Proposed behavior** Move scheduling into `server/src/services/heartbeat/scheduling.ts`. Keep the same public methods, admission rules, database claims, and execution behavior. The main file falls to 9,330 lines. **Reason and benefit** Give timer ticks, issue-monitor recovery, and session-goal recovery one clear module boundary. **Breaking changes** None intended. Existing heartbeat exports and method signatures remain available. ## What Changed - Add `createHeartbeatScheduling` with 11 explicit dependencies. Construction does not query the database or start timers. - Move timer ticks, issue-monitor dispatch and recovery, and session-goal recovery into the new module. - Keep queue admission, heartbeat policy parsing, and atomic timer claims in `queue.ts`. - Share the unchanged goal-control parser and issue-link formatter with their existing callers. - Add 20 focused tests. Document the module boundary in `doc/DEVELOPING.md`. ## Verification - 122 tests pass across 11 files: 20 new scheduler tests, 52 existing scheduling integration tests, and 50 adjacent queue, goal, native-selection, and task-drain tests. - New database tests cover cross-instance monitor claims, replacement schedules, deferred and failed admission, native ownership, archived companies, timer claim loss, worktree cutoffs, agent eligibility, goal recovery keys, pending-action deduplication, and invalid-action persistence. - AST comparison confirms 20 unchanged function bodies, unchanged remaining service statements, unchanged `executeRun`, and all 140 public exports preserved. - Workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. Greptile is 5/5 on commit `89a1a3b3aa428949eba349af5c1618391f97a19c`. All CI checks pass on that commit. The full test suite runs in CI; local verification was targeted. - The first CI attempt hit a 15-second timeout in the first test of unchanged `plugin-scoped-api-routes.test.ts`. All nine tests pass locally in 6.6 seconds. That shard passed on its one retry without code changes. - Run the new tests with `pnpm exec vitest run server/src/services/heartbeat/scheduling.test.ts --maxWorkers=1`. ## Risks - Incorrect wiring could bypass suppression, duplicate a monitor wake, or lose a replacement schedule. Database tests cover these boundaries and existing integration tests still call the public heartbeat service. - This PR moves the current behavior without changing scheduling policy, schema, provider execution, or cancellation ownership. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code editing, and command execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 15:09:11 -05:00
DottaandPaperclip	be1bc5d494	fix(runner): select bundled Linux assets for remote OpenCode (#15721) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native Runner can execute OpenCode in a remote sandbox. > - Installed releases contain a Linux daemon and the identity of their provider pack. > - OpenCode did not select those assets by default. Cursor did. > - The upload fallback also compared the verified Linux daemon with the controller platform. > - This pull request uses the packaged remote assets for OpenCode and corrects that platform check. > - Users of a matching Paperclip package and Daytona image can start OpenCode without artifact path overrides. ## Linked Issues or Issue Description Refs #15702. Refs #15422. **What happened?** Remote native OpenCode asked for `PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH` even when the installed release contained the expected pack identity. The daemon upload fallback could then reject a verified Linux daemon when the controller ran on macOS. **Expected behavior** OpenCode should select the release's Linux daemon and provider-pack identity. The controller must still verify the image assets before launch. **Steps to reproduce** 1. Run a complete installed Paperclip release on macOS. 2. Assign native OpenCode to a Daytona environment with the matching Paperclip image. 3. Start a task without remote artifact path overrides. ## What Changed - Select the bundled Linux daemon and provider-pack identity for remote OpenCode. - Skip the controller-platform comparison when the Linux daemon came from the verified bundled selector. - Add regression coverage for OpenCode and Cursor. Make the missing-identity fixture independent of local build outputs. - Document the matching package and image requirement, plus source-development setup. ## Verification - Red: both new regression cases failed before the fix. - Green: the affected runtime suites passed all 686 tests on the rebased head. - `pnpm -r typecheck` and `pnpm build` passed before the rebase. - Full local tests were attempted, then stopped after failures outside this change. That run overlapped a rebase. The clean rerun of the AI connection fixture passed all 111 tests. The affected runtime suites also passed after rebase. All latest-head CI and security checks passed: 54 successful checks and two expected skips. - A real OpenCode/OpenRouter DeepSeek task completed in an isolated Daytona workspace on macOS. Durable bash receipts confirm `df -h` and `uname -s; uname -m`. The result was posted, the task completed, and sandbox cleanup succeeded. - The live test used verified remote development assets from the official image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`. No remote artifact path overrides were set. This was not a new complete release assembly. - The low-trust test policy kept output quarantine. The operator permitted only this disposable test agent's OpenRouter credential binding. ## Risks - A second real external-origin retry executed successfully and saved its answer, but its completion report used unsupported evidence references. Its task stayed in progress. This PR fixes startup; completion-evidence guidance requires a separate fix. External voice playback was not verified. - A plain Daytona sandbox image lacks the provider pack. It still needs a verified full-pack override. - Source-development launchers must provide the remote build assets. This PR does not publish a new image or create a complete release assembly. - Explicit artifact overrides retain precedence. Manifest, executable, package, bridge, and provider-asset checks remain in place. ## Model Used OpenAI GPT-6 through Codex. The exact deployment model ID and context window are not exposed in this session. Capabilities used: reasoning, code editing, shell commands, and browser testing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merging Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 14:30:49 -05:00
DottaandPaperclip	6080daf262	fix(runner): start native OpenCode without postinstall (#15702) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner starts a qualified OpenCode runtime for native tasks. > - The default launch path uses a file populated by OpenCode's postinstall script. > - Package managers can skip that script and leave a text placeholder. Native task startup then fails with `spawn ENOEXEC` before the model request. > - This pull request resolves the declared platform package directly and includes the pinned runtime in the published server dependency graph. > - Users can run native OpenCode tasks after a normal install without global commands or manual dependency repair. ## Linked Issues or Issue Description **What happened?** Native OpenCode tasks fail at session startup when dependency postinstall is skipped. The Runner selects `opencode-ai/bin/opencode.exe`, which contains a text placeholder. The published server also lacks the OpenCode runtime dependency used by its vendored Runner. **Expected behavior** Paperclip starts the qualified platform binary installed by the package manager. macOS uses a Mac binary. No manual postinstall command is needed. **Steps to reproduce** 1. Install dependencies with lifecycle scripts disabled. 2. Configure a native OpenCode agent with an OpenRouter model and valid credentials. 3. Assign a task. The previous default fails before any model request. Related work: #15682 fixes HTTP transport registration. #15422 includes compatible OpenCode resolution in a broader agent setup change. This PR isolates the runtime repair for current users. ## What Changed - Resolve pinned OpenCode platform packages through the module dependency graph in source, standalone, and vendored server layouts. - Check package identity, version, platform, executable containment, and execute permission. Preserve verified executable launch profiles. - Use the resolver for new and resumed native sessions. Reject missing remote pack executables before controller dependency lookup. - Declare `opencode-ai@1.18.34` as a published server dependency. Leave platform selection to the consumer's package manager. - Check remote paths for the selected provider only. An unused OpenCode path must not reject a Pi run on macOS. - Add skipped-postinstall and invalid artifact regressions. Document automatic runtime selection. ## Verification - Regression tests fail with the previous default path and pass with platform resolution. - Source, standalone, and vendored layout fixtures cover Apple Silicon, Intel Mac, and Linux x64 baseline packages. - A fresh npm install with `--ignore-scripts` resolves the macOS executable and reports `1.18.34`. The placeholder remains unchanged. - Live browser test on Apple Silicon: a native OpenCode/OpenRouter/DeepSeek task ran shell arithmetic, created and uploaded a verified 23-byte file, and completed. A follow-up ran shell arithmetic and completed using the same session. Both runs committed successfully while the wrapper remained broken. - `pnpm -r typecheck` and `pnpm build` passed. All 269 affected Runner tests and 8 server packaging tests passed. `pnpm test:run` encountered 15 initial database-heavy failures; all 15 passed on an isolated group rerun. The redundant local run was then stopped after the matching server and workspace CI suites passed. All final-head checks passed (56 passed, 2 skipped); its unrelated chat-retry browser assertion passed on one rerun. An additional local repeat could not start its temporary server within the startup timeout. Greptile reviewed the final head at 5/5 with no open findings. ## Risks - Native runtime resolution requires the pinned optional platform package. Installations that omit optional dependencies fail with an explicit runtime error. - Supported native targets remain Apple Silicon, Intel Mac, and Linux x64. This change does not qualify another platform. - Published server installations now install the OpenCode dependency. Remote provider packs retain their verified executable paths and digests. - No schema changes or agent configuration migration. - Repository policy assigns `pnpm-lock.yaml` to CI. PR CI refreshes dependency resolution before frozen installs, and the master workflow updates the tracked lockfile. This PR leaves the tracked lockfile unchanged. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The session does not expose the exact GPT-6 variant or context window. - Live verification used OpenRouter `deepseek/deepseek-v4-flash-0731`; it did not author this patch. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 11:46:15 -05:00
DottaandPaperclip	0ca4b7c4c0	refactor(heartbeat): extract run cancellation and cleanup (#15685) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service admits, executes, and settles agent runs. > - Earlier extractions separated workspace, preparation, state, retry, recovery, queue, and lifecycle handling. > - Cancellation and resource release still live inside the main service. > - These operations share execution ownership and saved-comment continuation rules. > - This pull request moves them into one run-control module and adds boundary tests. > - Reviewers can inspect cancellation and cleanup separately from adapter execution. ## Linked Issues or Issue Description **What existing behavior does this improve?** The structure and testability of heartbeat cancellation and cleanup. **Subsystem affected** server/ — orchestration services. **Current behavior** Stop, pause and budget cancellation, environment lease release, and saved-comment delivery live in `heartbeat.ts`. **Proposed behavior** Keep these operations in `server/src/services/heartbeat/run-control.ts`. Bind their database and existing callbacks with `createHeartbeatRunControl`. **Reason and benefit** This removes 1,115 lines from `heartbeat.ts`, leaving 10,443 lines. The new module has 1,360 lines. Related cancellation and cleanup rules stay together in one file. **Breaking changes** None. Function bodies, service methods, public helpers, status conditions, and side effects keep their behavior. **Additional context** Continues the merged lifecycle extraction in #15672. Related cancellation work: #15212. Searches of open and closed issues and PRs found no duplicate run-control extraction. This does not add a roadmap feature. ## What Changed - Move run cancellation, pause and budget cancellation, lease release, issue-lock release, and saved-comment resumption into `heartbeat/run-control.ts`. - Supply lifecycle, queue, recovery, and cleanup callbacks through an explicit dependency interface. - Keep executor and cancellation maps shared by all service instances. Preserve callback construction order with forwarding functions. - Keep the existing helpers available from `heartbeat.ts`. - Add 15 tests for inert construction, cancellation gates, shared Stop barriers, failed termination, warm-resource retention, pending wake batches, project scope, and stale budget enforcement. - Document the module boundary in `doc/DEVELOPING.md`. ## Verification - Before extraction: **298 tests passed across six existing cancellation and cleanup suites**. - After extraction: **313 tests passed across seven files**, including the 15 new module tests and real PostgreSQL coverage. ```sh pnpm exec vitest run server/src/services/heartbeat/run-control.test.ts server/src/__tests__/heartbeat-native-runner-cancellation.test.ts server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts server/src/services/run-cancellation.test.ts server/src/services/adapter-execution-control.test.ts server/src/services/explicit-native-continuation.test.ts server/src/__tests__/native-cancellation-request.integration.test.ts ``` - Wider queue, recovery, reassignment, comment delivery, and native cancellation verification: **139 tests passed across nine files**, including the final module tests. The recovery suite first hit its 20-second database setup timeout while local builds were running. After removing verified orphaned PostgreSQL headers, its isolated run passed **380 of 381 cases**. One teardown assertion exceeded its one-second wait for a successor to settle; that exact case passed in a filtered rerun. Combined focused coverage is **818 distinct passing tests across 16 files**. The module suite appears in both runs. ```sh pnpm exec vitest run server/src/services/heartbeat/run-control.test.ts server/src/__tests__/heartbeat-comment-wake-batching.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-native-cleanup-admission.test.ts server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts server/src/services/heartbeat/queue.test.ts server/src/services/heartbeat/recovery.test.ts server/src/services/heartbeat/retries.test.ts server/src/services/heartbeat-stop-metadata.test.ts server/src/services/native-runtime/native-cancellation-request.test.ts ``` ```sh pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts -t "does not adopt unrelated queued comments for a non-coalescing recipient after Stop" ``` - Full `pnpm -r typecheck` and `pnpm build` passed on head `b26efd5f9de390ef8a675f84980f49f7fb7799c2`. - Full local `pnpm test:run` was started on this head and stopped after the complete CI suite passed. It did not finish and is not counted as a full local pass. - All **54 final-head checks passed**; two optional Storybook checks were skipped. This includes the full general and serialized test matrix, browser tests, typechecks, build, Runner verification, and canary dry run. The complete recovery suite passed in CI. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37945389261). - Greptile completed on head `b26efd5f9de390ef8a675f84980f49f7fb7799c2` with **5/5 and no actionable findings**. There are no review threads or merge conflicts. - Structural comparison confirms all 21 moved function bodies, the cancellation options type, remaining service logic, and all 140 existing public exports are preserved. ## Risks Moving closures can break callback construction order or shared Stop barriers. The factory starts no work and receives the same process-wide maps and existing callbacks. Tests cover concurrent Stop callers and failed termination. Native authority, status predicates, provider receipts, cleanup gates, and budget enforcement keep their original bodies. This PR has no schema or API changes. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 09:59:42 -05:00
DottaandPaperclip	49bbfe3092	fix(server): register native runner transport in HTTP startup (#15682) ... Register native Runner transport in the shared HTTP server constructor. Add startup regression tests and custom-launcher guidance. Verified with live OpenCode/OpenRouter tasks and complete CI coverage. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-09 09:53:42 -05:00
DottaandPaperclip	0b80ea17ef	refactor(heartbeat): extract run lifecycle handling (#15672) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service admits, executes, and settles agent runs. > - Earlier extractions separated workspaces, preparation, state, retries, recovery, and queue admission. > - Run status, progress, liveness, and completion handling still live in the main service. > - These operations share status predicates, event writes, and completion policies. > - This pull request moves them into one lifecycle module and adds boundary tests. > - Reviewers can inspect lifecycle policy separately from adapter execution. ## Linked Issues or Issue Description **What existing behavior does this improve?** The structure and testability of heartbeat run lifecycle handling. **Subsystem affected** server/ — orchestration services. **Current behavior** Run status writes, progress, events, liveness, completion handoffs, issue-comment finalization, and runtime settlement live inside `heartbeat.ts`. **Proposed behavior** Keep these operations in `server/src/services/heartbeat/run-lifecycle.ts`. Bind the database and reporting, recovery, and wakeup callbacks with `createHeartbeatLifecycle`. **Reason and benefit** This removes 2,682 lines from `heartbeat.ts`, leaving 11,558 lines. The lifecycle module has 2,982 lines and keeps related policy together in one file. **Breaking changes** None. Existing function bodies, service methods, public exports, status conditions, and side effects keep their behavior. **Additional context** Continues #15626 and the earlier merged heartbeat extractions. A search of open and closed issues and PRs found no duplicate lifecycle extraction. This does not add a roadmap feature. ## What Changed - Move status transitions, run events and progress, liveness classification, completion handoffs, plan-resume failure reporting, issue-comment finalization, and runtime/cost settlement into `heartbeat/run-lifecycle.ts`. - Supply reporting, recovery, and admission callbacks through an explicit dependency interface. Preserve construction order with forwarding callbacks. - Keep adapter execution, cancellation, and terminal telemetry reporting in the main service. - Add 16 tests for inert construction, continuation gates, late progress, status races, native ownership, usage metadata, revoked wakes, event sequencing, and redaction. - Update the existing publication-order source assertion to read the extracted event writer. Keep its persist-before-publish check. - Document the lifecycle module boundary in `doc/DEVELOPING.md`. ## Verification - Before extraction: 87 tests passed across seven existing suites. - After extraction: 103 tests passed across eight files, including the 16 new module tests and real PostgreSQL coverage. ```sh pnpm exec vitest run server/src/services/heartbeat/run-lifecycle.test.ts server/src/__tests__/heartbeat-run-status-payload.test.ts server/src/__tests__/heartbeat-run-event-sequencing.test.ts server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts server/src/__tests__/heartbeat-cost-accounting.test.ts server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts server/src/__tests__/run-liveness.test.ts ``` - Additional recovery, cancellation, plan-resume, summary, and handoff coverage: **595 tests passed across 14 files**. The module suite appears in both runs. With the publication-order suite below, combined focused coverage is **694 distinct tests across 22 files**, with no skips. ```sh pnpm exec vitest run server/src/services/heartbeat/run-lifecycle.test.ts server/src/services/heartbeat/recovery.test.ts server/src/services/heartbeat/retries.test.ts server/src/services/heartbeat/queue.test.ts server/src/services/recovery/successful-run-handoff.test.ts server/src/services/recovery/review-path-recovery.test.ts server/src/services/heartbeat-run-runtime-status.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-native-runner-cancellation.test.ts server/src/__tests__/heartbeat-native-cleanup-admission.test.ts server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts server/src/__tests__/heartbeat-runtime-state.test.ts server/src/__tests__/heartbeat-run-summary.test.ts server/src/__tests__/heartbeat-context-summary.test.ts ``` - The publication-order and lifecycle suites passed together: **28 tests across two files**. The first CI run exposed an assertion that still read the old event-writer source path. That assertion now reads the lifecycle module. ```sh pnpm exec vitest run server/src/services/chat-publication-reconciliation.test.ts server/src/services/heartbeat/run-lifecycle.test.ts ``` - After resolving the import conflict with the Pi Runner integration and rebasing onto `57e977be7`: **44 tests passed across four files**, including the updated source assertion, lifecycle module, run-status payloads, and accepted-plan workspace refresh. ```sh pnpm exec vitest run server/src/services/chat-publication-reconciliation.test.ts server/src/services/heartbeat/run-lifecycle.test.ts server/src/__tests__/heartbeat-run-status-payload.test.ts server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts ``` - Full `pnpm -r typecheck` and `pnpm build` passed on final head `bca87df089c541450932f6b678ca776942d267b2`. Full local `pnpm test:run` was restarted on this head and stopped after the entire CI suite passed. It did not finish and is not counted as a full local pass. - The earlier full local run had one transient public MCP Cloud bootstrap failure. All **82 public MCP tests passed in isolation**. The first local run was stopped before the source-assertion fix and rebase. - All **54 latest-head checks passed**; two optional Storybook checks were skipped. One general-test shard passed all 1,314 tests but failed during runner temporary-directory cleanup with `EACCES`. One rerun passed. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37940997283). - Greptile completed on final head `bca87df089c541450932f6b678ca776942d267b2` with **5/5 and no actionable findings**. There are no review threads or merge conflicts. - Structural comparison confirms 63 module function bodies and ten declarations match the originals. Remaining service function bodies and all 140 public exports are preserved. ## Risks Moving closures can break callback construction order or status settlement. The factory receives explicit callbacks and starts no work during construction. Compare-and-set predicates, native ownership holds, revoked-wake guards, event sequencing, redaction, and completion policies keep their original bodies. This PR has no schema or API changes. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 09:26:45 -05:00
DottaandPaperclip	f6d2a6b79e	refactor(heartbeat): extract queue admission and wakeup dispatch (#15626) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service admits and dispatches agent runs. > - Earlier extractions separated workspaces, preparation, state, retries, and restart recovery. > - Queue admission and wakeup dispatch still occupy more than 4,200 lines in the main service. > - Their transaction, budget, ownership, and concurrency checks must stay together. > - This pull request moves those operations into one queue module and adds boundary tests. > - Reviewers can inspect queue policy separately from adapter execution. ## Linked Issues or Issue Description **What existing behavior does this improve?** The structure and testability of heartbeat queue admission and wakeup dispatch. **Subsystem affected** server/ — orchestration services. **Current behavior** Wake coalescing, queued claims, dispatch, daily limits, and native wake intents live inside `heartbeat.ts`. **Proposed behavior** Keep their behavior in `server/src/services/heartbeat/queue.ts`. Bind lifecycle callbacks and shared ownership with `createHeartbeatQueue`. **Reason and benefit** This removes 4,243 lines from `heartbeat.ts`, leaving 14,201 lines. The queue module has 4,576 lines and keeps related policy together in one file. **Breaking changes** None. The service methods, public exports, database transactions, locks, and execution gates keep their existing behavior. **Additional context** Continues #15622 and the earlier merged heartbeat extractions. A search of open and closed PRs found no duplicate queue extraction. Related #15600 adds a new experimental scheduler; this change preserves the current scheduler policy. This does not add a roadmap feature. ## What Changed - Move wake coalescing and batching, queue claims, daily caps, concurrency and priority checks, timer admission, queue resumption, and native status wake intents into `heartbeat/queue.ts`. - Pass lifecycle callbacks and the existing process-wide execution and wakeup sets into the factory. Preserve construction order with three forwarding callbacks. - Keep adapter execution, cancellation, and status writes in the service. - Add nine tests for inert construction, scheduling suppression, shared wake tracking, policy wiring, claim release, newer ownership, and budget gates. - Document the queue module boundary in `doc/DEVELOPING.md`. ## Verification - Before extraction: 133 tests passed across nine existing suites. - After extraction: **150 tests passed across 12 files**, including all nine new module tests and real PostgreSQL coverage. ```sh pnpm exec vitest run server/src/services/heartbeat/queue.test.ts server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts server/src/__tests__/heartbeat-start-lock.test.ts server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts server/src/__tests__/heartbeat-scheduling-suppression.test.ts server/src/__tests__/heartbeat-comment-wake-batching.test.ts server/src/__tests__/heartbeat-running-followup.test.ts server/src/__tests__/heartbeat-dependency-scheduling.test.ts server/src/__tests__/heartbeat-auto-checkout.test.ts server/src/__tests__/heartbeat-archived-company-guard.test.ts server/src/__tests__/heartbeat-task-drain-admission-release.test.ts server/src/__tests__/heartbeat-worktree-suppression.test.ts ``` - Additional control and recovery coverage: **390 tests passed across five files**. Combined focused coverage is **540 passing tests across 17 files**, with no skips. ```sh pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-native-cleanup-admission.test.ts server/src/__tests__/heartbeat-native-status-context.test.ts server/src/__tests__/heartbeat-task-drain.test.ts server/src/services/chat-control-admission-retry.test.ts ``` - Full `pnpm -r typecheck` and `pnpm build` passed. - Full local `pnpm test:run` was started and then stopped after the full CI suite passed. It did not finish and is not counted as a full local pass. - All **54 current-head checks passed**; two optional Storybook checks were skipped. This includes the full general and serialized test matrix, typechecks, build checks, Runner verification, and canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37847175452). - Greptile completed on head `cc0001961a052a716d81c136929f89312e0b5dee` with **5/5 and no actionable findings**. There are no review threads or merge conflicts. - Structural comparison confirms 31 moved function bodies and eight declarations match the originals. Remaining service function bodies and all 140 public exports are preserved. - An initial local run skipped database tests because the host had exhausted its PostgreSQL shared-memory slots. After verified stale segments were cleared, all 150 focused tests ran and passed with no skips. ## Risks Moving closures can break shared ownership or callback construction order. The factory receives the original sets and explicit callbacks. It starts no work during construction. Transaction boundaries, compare-and-set predicates, budget gates, cleanup holds, and lock ordering stay intact. This PR has no schema or API changes. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 07:59:19 -05:00
Devin FoleyandPaperclip	9b624a110e	fix: checkpoint database backups before idle sleep (#15629) ... ## Thinking Path > - Paperclip coordinates agent work and preserves company state. > - Hosted instances can stop compute when admission and all work sources are quiet. > - The periodic database backup timer currently blocks every automatic sleep. > - Disabling backups would remove recovery points. > - This change offers an explicit final-backup checkpoint before sleep. > - A verified archive and restart marker preserve recovery while compute is stopped. ## Linked Issues or Issue Description **What happened?** An otherwise idle instance with automatic backups enabled always reports background work. Operators cannot reclaim idle compute without disabling those backups. The work inventory also queries the database before checking known local blockers. **Expected behavior** An operator can enable checkpoint mode. The server may authorize sleep only after all other work is quiet and a fresh backup is verified under the same owned hold. Backups resume on restart with the existing retention policy. **Steps to reproduce** Acquire a bounded idle drain on an instance with no application work and automatic backups enabled. Read its owned idle safety report. It reports present even when all other work is clear. With this change and `PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1`, a successful final backup can permit sleep; backup failures still refuse it. **Paperclip version or commit** Reproduced from master at 4265cb3a2b. **Deployment mode** Hosted single-process instances with persistent backup storage and serialized sleep/wake operations. Searched open backup and sleep PRs and issues. Related: #15522 and #15599 establish owned idle holds and plugin drains. #15358 proposes general backup health/retention validation; this change verifies only opted-in sleep checkpoints and their restart catch-up. ## What Changed - Add an off-by-default checkpoint flag. Keep automatic backups enabled. - Reject known local blockers before database work. - Finish a new backup during each owned safety check, including final revalidation. - Verify the complete gzip archive and sync its file and directory before pruning older recovery points. - Use unique checkpoint filenames so repeated checks cannot replace an earlier verified archive. - Keep backup work counted through dump, validation and persistence. Changed owners, expired holds, concurrent work and errors refuse sleep. - Persist a restart marker before the dump. Attempt a fresh backup on restart before idle eligibility, then resume the normal timer. Only a verified fresh backup clears the marker. - Restrict managed checkpoint reads to verified Cloud control actors. Tenant instance-admin elevation does not grant backup authority. - Keep archive verification outside engine fallback so failures retain their original cause. - Document that recurring duplicate archives pause while asleep. The final checkpoint and historical archives remain on persistent storage; retention does not prune while asleep. ## Verification - Focused idle, spool, backup and route checks: 137 tests passed, including restoration into a separate PostgreSQL database. - Backup library and checkpoint tests: 15 passed, including rejection before historical backups can be pruned. - `pnpm -r typecheck` and `pnpm build` passed. Follow-up server typecheck and database build cover the verification callback. - Review follow-up: 51 backup/restore/startup tests, 122 route/safety tests, 32 signed-control tests, and server typecheck passed. Startup cases exercise both flag states and retry after failure. The final test-cleanup adjustment passes all 35 startup tests. - The full local run hit four ancestor-directory skill-fixture failures and one tool-route failure. All five passed from an isolated checkout at 949210 (3 targeted chat/tool checks and the complete 39-test email suite). The redundant local run was stopped after final-head hosted CI passed. No full local pass is claimed. - Added-line secret/private-reference review and `git diff --check` passed. - [Final-head hosted CI](https://github.com/paperclipai/paperclip/actions/runs/37853380365) passed, including typecheck, build, all general and serialized test shards, all eight E2E shards and the canary dry run. Apex is 5/5 on 5b1de388ef, with no new findings and all review threads resolved. The branch is conflict-free. Live staging activation awaits merge and a canonical image; no production configuration or deployment changed. ## Risks - The flag defaults off. This is for a single process whose host controls every writer and preserves its backup volume across sleep. Independent database writers require a different backup owner. - Safety reads perform a backup only after a valid owned drain and all other checks pass. Slow backups can outlive the caller timeout; they remain counted until their work settles and cannot authorize sleep after hold expiry. - Initial and final reads each take a fresh backup. Hosts must allow sufficient request time while retaining shutdown headroom. - Retention settings are unchanged. Sleeping instances stop producing duplicate hourly archives and retain their last checkpoint. Filesystem data, encryption keys and off-provider disaster recovery remain separate responsibilities. - No schema migration. Disable the checkpoint flag to restore the blanket backup blocker; automatic backups and an existing restart marker remain active. ## Model Used OpenAI Codex, GPT-6. Used reasoning, repository inspection, code editing and command execution. The runtime did not expose an exact model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal identifier - [x] Focused and isolated local checks pass; final-head hosted CI is green (local full-run limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 15:58:55 -07:00
DottaandPaperclip	04de3132c2	refactor(heartbeat): extract restart recovery and lease cleanup (#15622) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service coordinates agent runs and their recovery. > - The earlier refactors separated workspaces, run preparation, run state, and retries. > - Restart recovery and lease cleanup still occupy more than 2,000 lines in the main service. > - These operations share process ownership and database claims that must stay intact. > - This pull request moves that group into one recovery module and adds boundary tests. > - Reviewers can now inspect recovery separately from queue admission and run execution. ## Linked Issues or Issue Description **What existing behavior does this improve?** The structure and testability of heartbeat restart recovery and lease cleanup. **Subsystem affected** server/ — orchestration services. **Current behavior** Hot-restart adoption, native restart recovery, shutdown draining, orphan reaping, and lease cleanup live inside `heartbeat.ts`. **Proposed behavior** Keep the same recovery behavior in `server/src/services/heartbeat/recovery.ts`. Bind the database and lifecycle callbacks with `createHeartbeatRecovery`. **Reason and benefit** This removes 2,181 lines from the main service. The extracted module is about 2,400 lines. It keeps related recovery operations together in one file. **Breaking changes** None. The public service methods, database claims, cleanup fences, and ownership checks stay the same. **Additional context** Continues the merged heartbeat extractions in #15568, #15573, #15578, #15591, #15601, and #15617. A search of open and closed PRs found no duplicate recovery extraction. This does not add a roadmap feature. ## What Changed - Move hot-restart snapshots and adoption, native restart recovery, shutdown draining, orphan reaping, and lease sweeps into `heartbeat/recovery.ts`. - Pass lifecycle callbacks and the existing shared execution sets into the factory. Preserve module-wide cleanup single-flight state. - Set the service's existing shutdown flag through a callback at the same point in shutdown preparation. - Add 14 focused tests for construction, shutdown ordering, empty selective drains, ownership races, shared state, retry timers, and background execution tracking. - Wait for the existing execution-drain barrier in the Stop recovery test instead of polling for one second. - Document the module boundary in `doc/DEVELOPING.md`. ## Verification - New module and additional cleanup coverage: **57 tests passed** across five files. ```sh pnpm exec vitest run server/src/services/heartbeat/recovery.test.ts server/src/__tests__/heartbeat-native-cleanup-admission.test.ts server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts server/src/services/hot-restart.test.ts ``` - Final recovery coverage on the rebased branch: **514 tests passed across 11 files**, including all 14 new module tests. ```sh pnpm exec vitest run server/src/services/heartbeat/recovery.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts server/src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts server/src/__tests__/native-session-resumption.test.ts server/src/__tests__/heartbeat-task-drain-admission-release.test.ts server/src/shutdown.test.ts server/src/__tests__/heartbeat-native-cleanup-admission.test.ts server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts server/src/services/hot-restart.test.ts ``` - Full `pnpm -r typecheck` and `pnpm build` passed, including after rebasing onto master. - Full local `pnpm test:run` was started again after the rebase. It was stopped after the full CI suite passed. It did not finish and is not counted as a full local pass. An earlier local AI-connection run hit an HTTP-test timeout; all corresponding CI coverage passed. - Structural comparison confirms 28 moved functions and 15 moved declarations. The only function-body adaptation is the shutdown flag callback. Remaining service functions and all 140 public exports match master. - All **54 current-head checks passed**; two unrelated Storybook checks were skipped. This includes the full general and serialized test matrix, typechecks, build checks, Runner verification, and canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37843023706). - Greptile completed on head `01fe39818d483e05f940b9fe842162a08e9b9e8e` with **5/5 and no actionable findings**. There are no review threads or merge conflicts. ## Risks The main risk is breaking shared execution ownership or changing recovery order while moving closures. The factory receives the original execution sets and lifecycle callbacks. It does no work during construction. Database transactions, compare-and-set predicates, cleanup deadlines, and native ownership gates stay intact. This PR has no schema or API changes. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 16:15:01 -05:00
DottaandPaperclip	2a5f65c9ae	refactor(server): extract heartbeat retry scheduling (#15617) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat orchestration starts runs and recovers from supported failures. > - The heartbeat service still contains more than 22,000 lines after earlier extractions. > - Retry scheduling has a clear boundary, but it shares lifecycle callbacks with the service. > - This pull request moves retry scheduling into one module with explicit dependencies. > - It preserves the scheduling rules, database writes, and lock order. > - The benefit is a smaller service and focused tests for the retry boundary. ## Linked Issues or Issue Description **Current behavior** `server/src/services/heartbeat.ts` owns retry schedules, contention deferrals, due promotion, and retry-now requests. These operations sit among execution and recovery code. **Proposed behavior** Move these operations into `server/src/services/heartbeat/retries.ts`. Keep the public heartbeat API and existing helper exports. The service supplies its lifecycle callbacks and worktree cutoff. **Reason and benefit** Extract one complete area in a separate PR. This removes 1,641 lines from the main service without changing retry policy. Future retry changes can use a smaller module and direct tests. **Breaking changes** None. Existing imports, error-class identity, retry metadata, and API outcomes remain available. **Additional context** Refs #15601, the preceding run-state extraction. Related open retry changes: #14930, #13975, #15157, #12587, and #13276. This PR moves the existing implementation. Those policy changes remain separate. ## What Changed - Add `createHeartbeatRetries(db, dependencies)` for bounded retries, workspace and connection deferrals, shared-workspace holder checks, due promotion, and retry-now requests. - Keep status transitions, event publication, issue-lock release, plan-resume reporting, and worktree settings in the heartbeat service. - Preserve all 140 existing public exports. Re-export the same helpers, constants, and workspace-busy error class. - Add 14 tests for construction, callback order, cancellation races, concurrent scheduling, committed promotion, and retry-now scope. - Document the module boundary in `doc/DEVELOPING.md`. ## Verification - Existing focused suite before the move: 84 tests passed. - Expanded focused suite after the move: 243 tests passed across 11 files. Command: `pnpm exec vitest run server/src/services/heartbeat/retries.test.ts server/src/__tests__/heartbeat-retry-scheduling.test.ts server/src/__tests__/heartbeat-workspace-busy.test.ts server/src/__tests__/heartbeat-model-fallback-warning.test.ts server/src/__tests__/heartbeat-running-followup.test.ts server/src/__tests__/heartbeat-issue-rewake-throttle.test.ts server/src/__tests__/issue-scheduled-retry-routes.test.ts server/src/modules/run-dispatch` - After rebasing onto the latest master, 61 retry and plugin-idle tests passed across four files. - `pnpm -r typecheck` passed again on the rebased branch. - `pnpm build` passed again on the rebased branch. - `pnpm test:run` was started locally. The long serial run was stopped after full CI passed; it is not claimed as a complete local pass. - Full CI passed on commit `69e76f3eec75b0703570f1fce37ebfe9c63fa3ac`: 54 successful checks, two intentional Storybook skips, zero failures. This includes general tests, serialized server tests, browser tests, runner checks, typecheck, build, and canary verification. - The setup-timeout review fix passed all 14 module tests and the server typecheck. - Greptile completed a fresh review of commit `69e76f3eec75b0703570f1fce37ebfe9c63fa3ac`: 5/5, no open findings. - A structural comparison confirmed 22 unchanged moved functions and one unchanged three-line string reader, 16 unchanged declarations, one unchanged API body, and all 140 preserved exports. The remaining service body is unchanged after accounting for the new binding and removed declarations. ## Risks - A missing lifecycle callback or an eager call during construction could change retry behavior. The dependency contract and direct tests cover these boundaries. - Concurrent scheduling must keep the issue-first lock order and reuse one successor. The original transaction bodies are unchanged. PostgreSQL tests cover concurrent calls from separate factory instances. - Promotion must publish after commit and preserve worktree cutoffs. Direct tests cover both paths. - No schema, route, adapter, or retry-policy changes are included. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 15:38:48 -05:00
DottaandPaperclip	4fe45bf8fe	refactor(heartbeat): extract run retrieval and session state (#15601) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat runs retain task sessions, usage totals, and bounded run data. > - After the preparation extraction, the main service still has over 24,000 lines. > - Run reads and session operations share one database and have no dispatch dependencies. > - This pull request moves that group into one module and preserves its callers. > - The benefit is a smaller run engine and one place to maintain saved run and session state. ## Linked Issues or Issue Description **What existing behavior does this improve?** It improves the structure and test coverage of heartbeat run retrieval and session state. **Current behavior** `server/src/services/heartbeat.ts` has 24,033 lines on the merged base. It mixes run projections, session queries, resume rules, compaction, and usage helpers with execution and recovery. **Proposed behavior** Move that group into `server/src/services/heartbeat/run-state.ts`. The main file loses 1,770 lines and ends at 22,263 lines. Keep current behavior, public exports, and service methods. This continues #15568, #15573, #15578, and #15591. **Reason and benefit** One domain extraction makes useful progress toward a few manageable files. Run reads and saved session state can be reviewed together without searching the run engine. A search found no duplicate open extraction. Related session changes include #15142, #14661, and #14670; this PR does not apply their proposed behavior changes. **Breaking changes** None. Existing public helpers retain their identity. Service method names and results stay the same. ## What Changed - Move bounded run projections, encoding checks, and run/event reads into `heartbeat/run-state.ts`. - Move task session persistence, explicit resumes, reset rules, compaction, and usage/billing helpers into the same module. - Bind database operations through `createHeartbeatRunState(db)`. Construction does no database work. The encoding cache stays local to each service instance. - Keep execution order, status transitions, session-goal recovery, cost accounting writes, dispatch, and cancellation in `heartbeat.ts`. - Preserve all 140 exports. A syntax-tree comparison confirms that 43 moved function bodies, 22 declarations, and nine service method bodies are unchanged. The copied private string helper is unchanged. Surrounding orchestration is unchanged apart from the factory binding. - Add eight tests for legacy export identity, database binding, encoding cache isolation, scoped resumes, cumulative usage, compaction, and stale or cancelled conversation session writes. - Document the module boundary in `doc/DEVELOPING.md`. ## Verification - Before and after extraction, eight existing suites pass: 235 tests. They cover workspace/session helpers, runtime state, cost accounting, ledger attribution, task reset, timer reset, run lists, and run privacy. - The new module suite passes: eight tests, including five real PostgreSQL cases. Total focused coverage: 243 tests across nine suites. - Commands: `pnpm exec vitest run server/src/services/heartbeat/run-state.test.ts` and `pnpm exec vitest run server/src/__tests__/heartbeat-runtime-state.test.ts server/src/__tests__/heartbeat-cost-accounting.test.ts server/src/__tests__/heartbeat-ledger-billing-code.test.ts server/src/__tests__/heartbeat-task-session-reset.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts server/src/__tests__/heartbeat-timer-wake-session-reset-pf4.test.ts server/src/__tests__/heartbeat-list.test.ts server/src/__tests__/heartbeat-run-privacy-routes.test.ts`. - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm -r typecheck` and `pnpm build` pass. - The full local `pnpm test:run` did not complete and was stopped with SIGINT after it reported an OAuth scope health-check failure and a workspace-runtime timeout. All six OAuth variants and the workspace case pass in isolation. A single rerun of the 413-case tool-access suite passed the OAuth case but hit a different Notion callback timeout; that Notion case also passes in isolation. The full local suite is not claimed as passing. - Greptile is 5/5 with no findings or inline comments on head `2ca0908cfc112dd6ae26e2565cb3611b5bec52ba`. Its exact-head check completed successfully. - CI remains blocked on hosted-runner assignment after more than ten minutes. [`ci / Select trusted runner`](https://github.com/paperclipai/paperclip/actions/runs/37828454507/job/113487287178) is queued for `ubuntu-latest` with no runner assigned. All seven completed review/security checks pass; two optional Storybook checks are skipped. The full CI matrix has not started. It must complete before merge. ## Risks - Wiring errors could bind reads to the wrong database or share the encoding cache. Direct factory tests cover database and cache isolation. Real PostgreSQL suites cover query and transaction behavior. - Session writes must retain their issue row locks and generation fences. The original function bodies are unchanged. New database tests reject stale and cancelled conversation writes and clears. - Existing SQL_ASCII safeguards and bounded projections remain in place. No schema, API contract, migration, lockfile, or workflow changes are included. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 14:53:48 -05:00
DottaandPaperclip	0d57b9b98e	refactor(heartbeat): extract run preparation (#15591) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service prepares context and configuration before agent runs. > - The workspace extraction left the main file at about 27,000 lines. > - Preparation helpers form another large boundary with explicit inputs and one database dependency. > - This pull request moves those helpers into one module and preserves their callers. > - The benefit is a 24,000-line orchestration file and one place to maintain run preparation. ## Linked Issues or Issue Description **What existing behavior does this improve?** It improves the structure and test coverage of heartbeat run preparation. **Current behavior** `server/src/services/heartbeat.ts` mixes context and configuration preparation with execution, queueing, and recovery in 27,048 lines. **Proposed behavior** Move preparation into `server/src/services/heartbeat/run-preparation.ts`. The main file loses 3,048 lines and ends at 24,000 lines. Keep the existing imports and behavior. This continues #15568, #15573, and #15578. **Reason and benefit** A larger domain extraction makes useful progress toward a few manageable files. Context, identity, environment, and tool preparation can be reviewed together without searching the run engine. **Breaking changes** None. Existing public helpers and the configuration-incomplete error class retain their identity. ## What Changed - Move wake payloads, comments, attachments, skill mentions, adapter environment configuration, and MCP/tool setup into `heartbeat/run-preparation.ts`. - Bind issue context, pinned routine snapshots, organization rows, and responsible-user resolution through `createHeartbeatRunPreparation(db)`. - Keep factory construction free of database work. Keep queueing, dispatch, retries, cancellation, and execution order in `heartbeat.ts`. - Preserve all 140 existing exports. A syntax-tree comparison confirms that the 47 extracted function bodies and 25 moved declarations are unchanged. The private string normalizer is also unchanged. - Add six tests for legacy export identity, independent database binding, company scope, pinned routine configuration, wake author selection, company-owner fallback, and credential preflight. - Document the preparation boundary in `doc/DEVELOPING.md`. ## Verification - Passed: 183 focused tests across ten suites, including the six new tests. Command: `pnpm exec vitest run server/src/services/heartbeat/run-preparation.test.ts server/src/__tests__/heartbeat-context-summary.test.ts server/src/__tests__/heartbeat-agent-session-message.test.ts server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts server/src/__tests__/heartbeat-runtime-skills.test.ts server/src/__tests__/heartbeat-project-env.test.ts server/src/__tests__/heartbeat-responsible-user-invariant.test.ts server/src/__tests__/heartbeat-comment-wake-batching.test.ts server/src/__tests__/run-secret-redaction.test.ts server/src/__tests__/low-trust-red-team-routes.test.ts`. - One runtime-skills case reported a Postgres deadlock in its after-test `TRUNCATE` cleanup. A focused rerun passed both runtime-skills cases. The other nine suites passed in the combined run. - Passed: `pnpm -r typecheck`. - Passed: `pnpm build`. - Passed: all 54 GitHub checks on `8d0cb9a251b10be5735fa13035e8a31e50b132e7`. The two optional Storybook jobs were skipped. The complete CI test matrix is green. - The full local `pnpm test:run` was started and then stopped after the complete CI test matrix passed. It had no failures reported before stopping and did not complete locally. - Greptile: 5/5 on the same commit, with no inline comments or actionable findings. ## Risks The extraction crosses context, identity, credential, and tool-access module boundaries. Existing company filters, secret redaction, low-trust rules, and error classes stay intact. The new factory captures only the service database and does no database work during construction. Existing integration tests cover responsible-user authority, credential boundaries, MCP tokens, and quarantine. New tests cover the factory wiring and legacy identity. A syntax-tree comparison confirms that orchestration changed only to bind the extracted loaders. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 13:47:29 -05:00
Devin FoleyandPaperclip	317ed367d4	Handle incomplete task policies and preserve review limits (#15593) ... Apply the reviewed change for Handle incomplete task policies and preserve review limits. Validation: local typecheck/build and focused regression tests, passing exact-head CI, Greptile 5/5, and independent source review. The PR records full-suite evidence and any local environment limitations. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-08 10:20:32 -07:00
DottaandPaperclip	75918997a8	refactor(heartbeat): extract workspace preparation and resolution (#15578) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The heartbeat service prepares workspaces and dispatches agent runs. > - Its main file still has more than 30,000 lines after two small extractions. > - Workspace preparation forms a larger boundary with one database dependency. > - This pull request moves that code into one workspace module with focused tests. > - The benefit is a smaller orchestration file and one place to maintain workspace policy. ## Linked Issues or Issue Description **What existing behavior does this improve?** It improves the structure and test coverage of heartbeat workspace preparation. **Current behavior** `server/src/services/heartbeat.ts` mixes workspace preparation, run execution, and scheduling in 30,634 lines. **Proposed behavior** Move workspace preparation into `server/src/services/heartbeat/workspaces.ts`. Keep the public imports and run behavior stable. The main file loses about 3,540 lines in one extraction. Related extractions: #15568 and #15573. **Reason and benefit** A domain-level extraction makes useful progress toward a few manageable modules. Future workspace changes can be reviewed without searching the whole run engine. **Breaking changes** None. The existing public exports and workspace validation error class retain their identity. ## What Changed - Move managed checkout preparation, workspace validation and reuse, referenced project resolution, and session/workspace config freshness into `heartbeat/workspaces.ts`. - Bind run workspace resolution to the database through `createHeartbeatWorkspaceResolver(db)`. - Keep the checkout single-flight map at module scope so all callers share pending materialization. - Keep all 140 existing `heartbeat.ts` exports. The 84 moved function bodies and 53 other moved declarations are unchanged in a syntax-tree comparison. - Add seven tests for legacy export identity, database binding, issue selection, session fallback, concurrent checkouts, same-name repositories, and retry after failure. - Document the module boundary in `doc/DEVELOPING.md`. ## Verification - Passed: seven focused workspace suites, 264 tests. Command: `pnpm exec vitest run server/src/services/heartbeat/workspaces.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts server/src/__tests__/heartbeat-referenced-projects.test.ts server/src/__tests__/heartbeat-remote-referenced-projects.test.ts server/src/__tests__/heartbeat-workspace-branch-containment.test.ts server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts server/src/__tests__/heartbeat-project-env.test.ts`. - Passed: `pnpm -r typecheck`. - Passed: `pnpm build`. - Passed: all 54 GitHub CI checks on `14dec775c7fa83716432f73740bd697106fdef68`. The two optional Storybook jobs were skipped. The full CI test matrix is green. - The full local `pnpm test:run` hit a 15-second timeout in the first `public-mcp.test.ts` case. A focused rerun passed all 80 public MCP tests. The long local run was stopped after all CI test shards passed; it did not complete locally. - Greptile: 5/5 on the same commit, with no inline comments or actionable findings. ## Risks Module initialization and shared checkout state are the main extraction risks. Legacy exports retain the same function and class objects. The checkout map stays outside the database factory. Existing integration suites cover company boundaries, workspace containment, reuse, and referenced project authorization. New local Git tests cover concurrent materialization and failure cleanup. The orchestration body is unchanged except for the resolver factory binding. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 10:56:33 -05:00
DottaandPaperclip	87a7312cfd	refactor(server): extract heartbeat run-log formatting (#15573) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat orchestration records agent output and run events. > - The heartbeat service still contains more than 30,000 lines after the first extraction. > - Log formatting has fixed limits and does not write run state. > - These helpers form a small next step toward a more manageable heartbeat service. > - This PR moves them into the heartbeat folder without changing their function bodies. > - Direct boundary tests and existing caller tests check the output. ## Linked Issues or Issue Description Refs: #15568. **What existing behavior does this improve?** This improves the structure and test coverage of heartbeat run-log formatting. **Subsystem affected** server/ — orchestration services. **Current behavior** `heartbeat.ts` contains excerpt handling, payload size limits, and log chunk formatting beside run orchestration. **Proposed behavior** Move these helpers and their constants into `server/src/services/heartbeat/run-log.ts`. Keep the same function bodies, output, limits, and public exports. **Reason and benefit** Run-log formatting has its own small module and focused tests. This keeps the second extraction small enough to review on its own. **Breaking changes** None. The existing public import path and persisted output stay the same. Related run-log PRs: Refs: #6373, Refs: #8841. Those PRs change redaction behavior. This PR only moves existing formatting code. ## What Changed - Move 109 lines of helper functions and six constants into `heartbeat/run-log.ts`. - Move `appendExcerpt` and retain the existing public exports for `boundHeartbeatRunEventPayloadForStorage` and `compactRunLogChunk` in `heartbeat.ts`. - Add 17 direct regression cases for size and depth limits, cycles, shared references, immutable input, image omission, redaction order, excerpt tails, and UTF-8 boundaries. - Document both heartbeat extractions in `doc/DEVELOPING.md`. ## Verification - Before extraction, four focused files passed with 55 tests. - After extraction and the two new excerpt tests, the same four files passed with 57 tests. - Run `pnpm exec vitest run server/src/services/heartbeat/run-log.test.ts server/src/__tests__/heartbeat-run-log.test.ts server/src/__tests__/heartbeat-list.test.ts server/src/__tests__/redaction.test.ts`. - The original and extracted function blocks match byte for byte after adding the export keyword to `appendExcerpt`. - Existing tests still import the public helpers from `heartbeat.ts`. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - `pnpm -r typecheck` passed. - `pnpm build` passed. - The full suite passed in CI. The duplicate local `pnpm test:run` was stopped after CI passed; it did not complete locally. - All CI gates passed on commit `4e65c666e3b5b46162b3a528b37fba8902a1d200`: 54 passing checks, two skipped Storybook checks. - Fresh Greptile review of the same commit: 5/5, no actionable or inline findings. The PR has no merge conflicts. ## Risks - Low risk. Moving code can cause an import or build error. - The same redaction and adapter utility modules remain in use. - Database writes, current-user redaction, live event delivery, and run state remain in `heartbeat.ts`. - The event schema and payload output do not change. - No database, API, Telemetry, Observability, or UI contract changes are required. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 09:34:37 -05:00
DottaandPaperclip	51653becc2	refactor(server): extract heartbeat task markdown rendering (#15568) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat orchestration supplies each run with task context. > - The task Markdown renderer is inside a service with more than 31,000 lines. > - The renderer formats task data and does not write run state. > - It is a small first step toward a more manageable heartbeat service. > - This PR moves the renderer without changing its function body or public export. > - Direct tests and existing caller tests check the output before and after the move. ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves the structure and test coverage of heartbeat task context rendering. **Subsystem affected** server/ — orchestration services. **Current behavior** `buildPaperclipTaskMarkdown` occupies 352 lines inside `heartbeat.ts`. The service also owns run execution, scheduling, recovery, and cancellation. **Proposed behavior** Move the renderer to `heartbeat/task-markdown.ts`. Keep its function body and the existing `heartbeat.ts` export unchanged. Leave orchestration in place for this first PR. **Reason and benefit** Prompt formatting has its own small source file and direct regression tests. Reviewers can check this one extraction before a later refactor. **Breaking changes** None. The input type, rendered output, and existing import path stay the same. Related renderer changes: Refs: #4732, Refs: #14030. These PRs change prompt behavior. This PR only moves the current renderer. ## What Changed - Move the 352-line renderer to `server/src/services/heartbeat/task-markdown.ts`. - Import and re-export it from `heartbeat.ts`. Remove imports used only by the renderer. - Add seven direct regression cases. Cover empty context, ordered comment-only wakes, input preservation, nested code fences, ancestor limits, attachment-only wakes, and rejected plans. - Keep the renderer and its direct tests in `server/src/services/heartbeat/`. Document this folder as the home for relevant later extractions in `doc/DEVELOPING.md`. ## Verification - The focused suite passed before and after extraction, and after moving to the heartbeat folder: four files, 85 tests. - Run `pnpm exec vitest run server/src/services/heartbeat/task-markdown.test.ts server/src/__tests__/heartbeat-context-summary.test.ts server/src/__tests__/heartbeat-chat-task-link.test.ts server/src/__tests__/codex-local-execute.test.ts`. - The extracted function matches the original function byte for byte. The folder move only adjusts imports. - `node scripts/check-module-boundaries.mjs` passed. - `git diff --check` passed. - The initial extraction passed local `pnpm -r typecheck` and `pnpm build`. - The folder update passed `pnpm --filter @paperclipai/server exec tsc --noEmit` and `pnpm --filter @paperclipai/server build`. - The initial extraction passed the full CI suite. Its duplicate local `pnpm test:run` was stopped after CI passed; it did not complete locally. - The folder update passed all CI gates on commit `e21e589b379d7ca2fae16c1dcd91cf2f604fd03f`: 54 passing checks, two skipped Storybook checks. Three test shards passed after one retry following simultaneous runner shutdowns; those failures had no failed test assertions. - Fresh Greptile review of commit `e21e589b379d7ca2fae16c1dcd91cf2f604fd03f`: 5/5, no actionable or inline findings. ## Risks - Low risk. A moved module can change import resolution or expose an import cycle. - Existing caller tests still load the compatibility export from `heartbeat.ts`. - The same guidance constants and public task URL resolver remain in use. - No run-state writes, transactions, locks, shared process state, or cleanup paths moved. - No database, API, telemetry, or UI contracts changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-6 via Codex. The exact serving model ID and context window were not exposed in this session. Used reasoning, repository inspection, shell tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 09:06:04 -05:00
DottaandPaperclip	d66acb7ac1	feat: automate Slack bot app setup and installation (#15413) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat connectors give each agent a customer-owned bot and task-backed conversations. > - Manual Slack setup requires app creation and copying durable credentials. > - Operators need a shorter setup that an assisting agent can use safely. > - This pull request creates the app through Slack's Manifest API and installs it through OAuth. > - Durable registration state supports recovery without creating another app. > - A four-screen wizard, automatic avatar upload, and OAuth account linking reduce setup work. > - Connector settings and per-turn tool guidance support daily use after installation. ## Linked Issues or Issue Description **Subsystem affected** Native Slack bot setup, company secret storage, chat connector management, and agent tool guidance. **Problem or motivation** New Slack bots require manual app creation and copying a signing secret and bot token. Interrupted setup can create duplicate apps. The setup and management screens contain unnecessary controls. Agents also need guidance for native questions, files, thread replies, and governed Slack actions. **Proposed solution** Use a temporary app-configuration access token to create a customer-owned app. Save durable secrets in the vault. Bind OAuth to the initiating actor, company, endpoint, registration revision, scopes, and configured origins. Preserve manual and existing-app recovery. Link the installing user's account, send a welcome DM, and advance from saved server evidence. Keep request-URL recovery instructions available if automatic connection detection waits. **Alternatives considered** The Slack CLI adds installation requirements. Socket Mode changes transport. A shared Paperclip-owned app changes app ownership. These alternatives are outside this change. **Roadmap alignment** This extends existing chat connectors and secrets capabilities. Related public work: #14037 and #13954 cover Slack MCP prerequisites and user OAuth. No duplicate bot-registration PR was found. ## What Changed - Share one reviewed manifest builder between automatic registration and manual setup. - Add replay-safe migration 0318 and company-bound registration state with vault references and uncertain-creation recovery. - Add registration, installation, callback, and resume APIs with short-lived, single-use OAuth state. - Save installation credentials before downstream checks and preserve bot identity constraints. - Reduce automatic setup to four screens. Keep advanced app details, manual recovery, and existing-app setup. - Upload the agent avatar with the Paperclip dark background. Link the OAuth installer's account and send setup DMs. - Show agent and connector-owner avatars. Simplify settings, access, and conversation screens. - Discover joined Slack channels and enable them by default. Start a task from a bare mention and admit same-thread follow-ups. - Refresh Slack tool guidance each turn. Add native-form, file, approval, and delivery regressions plus manual model probe definitions and sanitized acceptance records. - Update deployment/database docs, OpenAPI, redaction, removal cleanup, production Storybook stories, and provider browser tests. - Merge current master and move the registration migration after its latest migration without rewriting published commits. The completed Slack success view intentionally has a single centered **Done** action and no **Save & exit**, as explicitly requested by the product owner. `DESIGN.md` records this exception; unfinished setup steps retain the aligned wizard footer. ## Verification - Passed after the master merge: repository typecheck, full build, Storybook build, design-token gates, module-boundary gates, and migration generation. - Passed: all 352 focused Slack deterministic tests and all 14 affected provider browser tests. Browser tests use controlled provider fixtures and a separate throwaway instance. - Passed on current head `c5d01e0e2`: the complete GitHub test matrix (general server, chat, all workspaces, serialized server, and Runner), all eight browser shards, typecheck/release registry, build, canary dry run, security checks, and policy gates. There are 52 passing checks and no pending or failing checks. - Greptile completed on the exact current head with 5/5 and no actionable findings or open review threads. - Local repair verification passed 93 focused tests, including same-app reinstall after revocation and rejection of consent started before revocation, the AgentMail browser journey, and repository typecheck. Local build and Storybook build also passed. The redundant local full-suite rerun was stopped after the complete current-head CI matrix passed. - Real Slack setup and agent replies were exercised in the authorized isolated test drive during the setup iteration. - The ten additional model probes were attempted with legacy `codex_local`, `gpt-5.6-sol`: five passed, two failed, and three were partly verified. Native runtime is not qualified. See `server/src/services/connectors/slack/evals/2026-10-08-acceptance.md` for evidence and limits. - Passing model probes cover native forms, downloaded file bytes, bare mentions with thread replies, explicit posts/reactions, and saved approval denial. - The controlled uncertain-write probe found wrong delivery-check IDs. The canvas fallback attempt used an invented tool name. Search pagination/native search, a private-source denied-tool receipt, and distinct board/webhook origins remain unqualified. Reviewer path: enable Chat connectors, start Slack chat setup, select an agent, enter an app-configuration access token, and approve Slack installation. Send a message to the bot and confirm that setup advances to success. Inspect settings and allowed channels. See `doc/connections/SLACK-AUTOMATIC-SETUP.md` for deployment and recovery. ## Risks - Slack app creation has no provider idempotency guarantee. A timeout after dispatch stays uncertain until the operator checks Slack. - OAuth needs a stable public HTTPS board origin. Webhook ingress may use a separate configured HTTPS origin. Workspace policy can delay installation. - Migration 0318 can replay safely on instances that applied the earlier development migration. - OAuth installation now links the installer to the initiating Paperclip user. Identity checks and company access rules still apply. - Joined channels now enable bot responses by default. Linked-user authorization and per-action approval rules still apply. - Model behavior has the documented delivery-check and canvas fallback failures. A passing CI run does not establish that every model probe passed. - Removing the connection does not delete the customer's Slack app. No new first-party telemetry is added. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and browser verification. The runtime does not expose a more specific authoring model ID or context-window size. The live bot probes used OpenAI `gpt-5.6-sol` through `codex_local` in legacy mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 06:33:31 -05:00
DottaandPaperclip	d0f69670db	fix(runner): recover saved execution prompts after upgrades (#15518) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs save an immutable execution context for restart recovery. > - The context includes the prompt text, revision, and content hashes. > - The parser required that saved prompt to match the current release. > - A server upgrade could reject a valid saved run before provider recovery. > - This pull request validates and preserves the saved prompt snapshot. > - Routine prompt changes no longer need a catalog of past strings. ## Linked Issues or Issue Description **What happened?** A hot restart selected a dead native runner for same-run recovery. Reading its saved v5 execution input failed with `input.runtimeContext.prompt must match the fixed Paperclip prompt revision`. The new controller accepted only v6. **Expected behavior** Recovery uses the saved prompt and validates its content hashes. It preserves the same run and provider session without starting a duplicate turn. **Steps to reproduce** 1. Start a native run and save its execution input and provider checkpoint. 2. Change the fixed execution prompt in the server release. 3. Stop the runner and recover the saved run with the new controller. 4. Observe that the old parser rejects the saved prompt before provider recovery. **Paperclip version or commit** The v5-to-v6 prompt change was introduced in #15446. The defect also reproduces on current master before this fix. **Deployment mode** Source-built server with the native runner. Related work: #15446 added task-monitor guidance. The held prompt-size experiment in #15489 changes prompt wording but does not add recovery compatibility. ## What Changed - Read the prompt text and revision from the saved execution snapshot. - Treat the revision as non-empty metadata and preserve the exact saved bytes. - Validate the prompt SHA-256 and the aggregate context digest. - Keep fresh-run builders on the current prompt constants. - Test arbitrary saved prompts, malformed fields, altered text, stale hashes, and aggregate drift. - Test recovery parsing for Codex input versions v3-v5 and OpenCode, ACPX Pi, and Dot v6 inputs. - Extend the real-process restart suite with both the incident's v5 wire fixture and a prompt unknown to this release. - Run the restart recovery suite in the existing Rust-equipped PR lane, where its runner and fake-provider binaries are built. Verify complete, non-overlapping test coverage for PR, release, and local callers. - Document recovery from saved snapshots without a historical prompt catalog. ## Verification - Red: the new contract regressions fail against the catalog-based parser with the original prompt-validation error. - Green: 53 focused contract and materialization tests pass. - Red: the real-process unknown-prompt regression fails with master's original parser at the saved-input recovery read after process loss. - Green: all 15 real-process restart tests pass locally on the final branch. The saved-prompt cases keep the run and provider session, replace the PID, and record one `turn/start`. - Local repository `pnpm -r typecheck` and `pnpm build` passed after rebase on `89f09dad723766e5351953f0731b9aa5daada28d`. The 53 focused tests also passed on that head. - Red: the new test-roster checks fail against the old CI placement. - Green: all 26 test-scheduling checks pass after moving the restart suite. - CI ran all 15 restart recovery tests with no skips on final head `6719fc2bb7a31a0f72ea04c7e525a63dcc6f9105`. [Runner test job](https://github.com/paperclipai/paperclip/actions/runs/37764782334/job/113271464966). - Greptile scored 5/5 on that exact head with no actionable findings. - The complete CI matrix passed on final head `6719fc2bb7a31a0f72ea04c7e525a63dcc6f9105`: general/workspace tests, serialized server suites, both runner Vitest lanes, Rust and static checks, all browser shards, typecheck, build, and the canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37764782334). - There are no unresolved review threads or merge conflicts. - Reproduce focused tests with `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/contracts/runtime-context.test.ts src/contracts/native-execution.test.ts src/drivers/runtime-context-materializer.test.ts`. - Reproduce restart tests with `pnpm --filter @paperclipai/paperclip-runner build:rust` followed by `pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. They use temporary PostgreSQL, real runner processes, and a fake Codex provider. They do not use paid inference. ## Risks - The parser now accepts internally consistent saved prompt text that is absent from the current source. Inputs must come from trusted server persistence. Content hashes verify consistency; they do not authenticate authorship. - Existing execution-schema, ownership, checkpoint, provider, permission, and session-compatibility checks still apply. - This change validates the saved base prompt. It does not make all additional code-generated instruction strings versioned. - The process-level recovery proof uses Codex. Other provider coverage verifies the shared input parser and retained provider configuration. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code editing, and test tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 05:57:20 -05:00
DottaandPaperclip	bf9dbd18a8	fix(runner): recover incompatible Codex models before launch (#15519) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Runner prepares and saves each task's provider configuration before launch. > - A sandbox image can have a supported Codex CLI that is too old for the selected model. > - The current version check stops that task even when the image can run a similar model. > - This pull request selects a compatible model before it saves a fresh execution. > - The task continues with a visible warning, and recovery uses the saved effective model. ## Linked Issues or Issue Description - Refs #15053. This fixes the same CLI and model mismatch for fresh remote Runner tasks. It does not change subscription onboarding. - Refs #14721. This adds a bounded startup choice for Codex CLI compatibility. It does not add user-defined fallback chains or quota failover. - Related PR: #15399 added the model-specific CLI checks that this change preserves at launch. ## What Changed - Check the preinstalled remote Codex CLI before saving a fresh execution that uses a model with a verified CLI minimum. - Select the closest compatible older model in the same class, then the stable Runner default. Consider each candidate once. - Save the effective model before checkpoint selection. Keep the requested agent and task settings unchanged. - Add a task warning and a local `runner.model_fallback` run-log event with both models and the CLI version. - Share executable discovery with the launch verifier. Preserve explicit artifact and install settings, saved executions, and existing artifact checks. - Add regression tests and document the selection and warning behavior. ## Verification - Red: the three new provider-configuration regression cases failed before the fix. They kept the incompatible requested model. - Red: both rejected-probe cases failed before the review fix. They now defer to launch verification. - Green: targeted provider configuration, remote preflight, task warning, and launch-verifier tests pass (680 tests). - `pnpm -r typecheck` passes. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/37713759744) passes on `eb273fc32661526d90a864b42c73a7da61dd62da`: 47 successful jobs, including all general and serialized Vitest groups, browser shards, Runner verification, typecheck, build, and the canary dry run. - Local extended verification: three serialized shards pass (115 suites). HTTP route tests hit intermittent 15-second timeouts. The authorization suite passes on rerun (130 tests). The document suite passes on both master and the PR head in the same isolated setup (6 tests). The local general run was stopped after the equivalent full CI matrix passed. - `pnpm build` passes. - Server typecheck and build pass again after the probe-error review fix. - `git diff --check` and `pnpm check:module-boundaries` pass. - Example: request `gpt-6.1-sol` on Codex 0.158.0 to use `gpt-6-sol`; on 0.156.0, use `gpt-5.6-sol`. The warning names the requested model, effective model, and CLI version. - This PR has not been deployed to staging. ## Risks - A fallback can have different capabilities. The task warning makes the substitution visible. Later runs can use the requested model after the image CLI is updated. - Preparation adds two remote commands for models with a verified CLI minimum. Failed or invalid version probes retain the existing launch checks. - This only handles known CLI and model mismatches before a fresh provider launch. Authentication, capacity, and artifact failures keep their existing behavior. - No database migration is required. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. This session does not expose the exact deployment model ID, context window size, or reasoning setting. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 04:57:26 -05:00
Devin FoleyandPaperclip	5717523b9e	fix: retain the original repository for reused task workspaces (#15528) ... Apply the reviewed change for fix: retain the original repository for reused task workspaces. Validation: required local typecheck and tests, passing CI, Greptile 5/5, and independent review of the exact source head. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-07 19:52:30 -07:00
DottaandPaperclip	dd777f4b73	feat(dot): complete onboarding and expand governed Runner capabilities (#15414) ... ## Thinking Path > - Paperclip manages AI agents, their work, and their permissions. > - Paperclip Runner supplies the same admitted tool authority to each provider. > - The Dot provider in #15402 needs reliable onboarding and useful agent capabilities. > - An idle Dot could not start work, assign a human, read skills, or produce a workspace artifact. > - Pairing also relied on a second configuration save before ordinary admission could work. > - This pull request adds governed idle admission and shared Runner tools, and completes pairing atomically. > - Operators can test Dot with fresh data while keeping normal company, approval, budget, and run ownership checks. ## Linked Issues or Issue Description **Subsystem affected** Paperclip Runner, dedicated Dot MCP access, OAuth onboarding, experimental settings, and empty worktree startup. This PR builds on merged provider PR #15402. It reuses the merged MCP gateway from #14846 and assistant connection work from #14933 and #15380. **Problem or motivation** An idle Dot could see assignments but could not act on a conversation request until someone created a task first. Its Runner catalog could not assign tasks to humans or read pinned skills and workspace files. First-time OAuth discovery and pairing also needed browser fixes, and a completed pairing did not persist its binding reference on the agent. **Proposed solution** Keep Dot within the existing Runner. Admit a visible agent-authored intake task for idle requests. Add people, human assignment, cross-task, skill, and optional sandbox workspace tools through shared authority. Relay assigned app calls through the configured MCP gateway. Add lease renewal and follow-up references. Save pairing and its configuration revision atomically. Show prerequisites and provide a complete copy prompt. **Roadmap alignment** This extends the experimental provider in #15402. It uses the existing governed gateway, task model, skills, artifact path, and native Runner. It adds no separate execution subsystem. ## What Changed - Add a standalone OpenAI Dot agent choice with an independent experimental opt-in. It works with the general Runner option off. Require Assistant connections (MCP), authenticated sign-in, and public HTTPS for pairing. - Use Dot’s own option for company package import. Allow an unpaired Dot configuration to save after external billing acknowledgement; task admission still requires pairing. Prepare the shared dev binary for Dot-only opt-in. - Label saved Dot agents as OpenAI Dot. Use prerequisite-check copy in setup and runtime configuration. - Route Dot creation directly to pairing after explicit external billing acknowledgement. Hide local CLI, model, and harness setup for Dot. Preserve the shared Runner implementation and canonical API type. Other Runner providers still require their general opt-in. - Add an empty worktree option with fresh signing keys and no production data copy. - Fix public-client OAuth negotiation, discovery compatibility, and optional separate browser authorization origin. - Add one-use pairing consent preview, clear copied setup instructions, and atomic binding persistence and cleanup. - Add idle request admission with stable request IDs and normal scheduling, permissions, budgets, and task ownership. - Add identity and people discovery, human task assignment and reassignment, and authorized cross-task comments and documents. - Read assigned skill files from pinned manifests. Relay assigned app calls through the merged gateway without exposing credentials. - Add an off-by-default workspace bridge. Constrain paths and writes. Run commands in a deny-by-default OS sandbox with no network or injected credentials. Reserve mutations before effects and never blindly repeat uncertain work. - Add rolling lease renewal, task pagination, bounded operation limits, and deduplicated follow-up references without comment bodies in webhooks. - Add an off-by-default attachment reading setting. Restrict reads to files on the current assigned task. Verify size and hash, cache bounded verified copies per run, paginate text or binary bytes, and recheck live authority before returning. - Keep file grants operator-owned. Reject agent self-grants across configuration routes. Preserve attachment consent in create/import forms. Close generic API file bypasses while retaining current-run response snapshots and permitted uploads. - Keep provider limits explicit. Do not inherit a Dot binding, attachment permission, or workspace permission when hiring another agent. - Include the required Markdown format in cross-task document writes and validate the API title limit. Verify real creation and revision persistence. - Restrict command execution to Linux bubblewrap with descendant containment. macOS retains workspace file tools and artifact publishing, while refusing command calls. Explain the platform limit in setup. - Exclude Paperclip instance state from workspace files, uploads, artifact publication, and sandbox commands. Protect nested directories and case variants. Fail closed when the directory protection scan exceeds 4,096 directories. - Add static UI compression for slow public tunnels. Document setup, the complete tool inventory, and qualification limits. ## Verification - Merge preparation on `00ca2c75b` integrates merged base #15402 and master `fc6304dfe`. The ancestry commit preserves the reviewed follow-up source tree. The subsequent security fix excludes instance state from file tools, uploads, artifact publication, and sandbox commands. It preserves private task authorization and task monitors. It regenerates the combined tool catalog, seeded catalog digest, and protocol manifest. Workspace typecheck, full build, and UI token gates pass. All sixteen real Dot broker cases and 93 company import cases pass after the review fixes and creator-attribution test correction. The exported human-assignment catalog and Unicode page boundaries are also fixed. All ten catalog tests and nine workspace/skill bridge tests pass. All 35 workspace bridge and authority tests passed after the instance-state fix, including real macOS commands in that intermediate version. The subsequent document and descendant-containment fixes pass 44 focused tests across bridge, authority, and setup UI, with six Linux command cases skipped on macOS. Real cross-task documents pass the route validator and persist two revisions. macOS refuses command execution and does not advertise the tool. Full workspace typecheck and build, changed server/UI typechecks, and token gates pass again on the final commit. Greptile rates final head 00ca2c75b 5/5 and its completed check concludes success; all review threads are resolved. All 58 current-head checks are complete: 54 passed and 4 intentionally skipped. This includes full tests, typecheck, build, Runner, browser E2E, release verification, and Canary Dry Run. The older full local root attempt finished with 16,602 passes, 93 skips, and ten failures across two suites: it began before source edits and retained earlier imported implementations while loading later tests. Both suites pass a clean final-head rerun (25 passed, six Linux-only command cases skipped on macOS). This mixed-source full attempt is not a final-head full-suite pass; use the fresh CI evidence below. - Full workspace typecheck and build pass for the standalone Dot change on head `392d54f80`. UI token gates are clean. The full workspace typecheck passed before the final importer UI edit; the changed UI typecheck, build, and token gates pass again on the final commit. The server build passes for its unchanged final source. - Standalone selection, creation, settings, harness visibility, inventory, and runtime admission checks pass. OAuth onboarding and the real Rust/PostgreSQL broker suite pass 27 cases with the general Runner option disabled. Dot and MCP revocation still block pairing; other Runner providers remain disabled. Create, hire, conversion, and inherited-hire route regressions pass 68 cases. The runtime selection suite passes 20 cases. The setup UI suite passes 50 cases. Company import and dev binary checks pass 97 cases. Import UI checks pass 29 cases, including Dot-only selection, preservation of imported Dot agents, generic Runner fallback, canonical configuration serialization, and required billing acknowledgement. A read of the synthetic instance reports the native binary required with Dot enabled, the general rollout disabled, and no persisted native work. - The latest attachment, operator-consent, generic API file-guard, and bridge checks pass 80 tests. Cache and native lifecycle checks pass 550 tests; create/import builders pass 45 tests; attachment setup UI checks pass 16 tests. The real Rust/PostgreSQL Dot broker suite passes 15 cases. - Earlier authority, human assignment, pinned skill, confinement, cancellation, inheritance, onboarding, replay, OpenAPI, and gateway regressions pass their focused rechecks. Workspace writes reject concurrent stale hashes. - In the live empty test-drive, turn the general Runner option off and leave Dot and MCP on. Add agent shows a separate OpenAI Dot choice. Create rejects missing billing acknowledgement, saves the canonical Dot configuration, and opens pairing. The existing paired Dot remains ready and passes its prerequisite checks. No new plugin pairing was needed for this UI change. The final browser import preview keeps a Dot source agent as OpenAI Dot, falls back a generic Runner agent while its switch is off, and offers Dot independently. - Real Dot completed OAuth, signed MCP Events readiness, and an event-only assigned document task in an empty synthetic instance. Pairing saved its binding without a second configuration save. - Real Dot verified human assignment, cross-task comments and documents, pinned skill reading, sandbox commands, lease renewal, follow-up input, and a downloaded artifact whose bytes and hash matched its receipt. An idle conversation request created an intake, created a task for its human owner, continued after a definite missing-file read error, and finalized Done with exit code 0. - After operator approval, real Dot read a synthetic assigned-task attachment and wrote its file-only random proof into an agent-authored document. Its first test required accepting review because the existing document tool removed the final newline. - On head `2626c8f9b`, real Dot read a 13,849-byte synthetic attachment in two pages, used `expectedSha256` on page two, saved exactly its final random marker, verified readback, and finalized Done with exit code 0. A direct inbox check was required for this attachment qualification; it does not claim event-only delivery. - Previous head `392d54f80` passes all 55 checks (53 passed, two intentionally skipped), including the full test, typecheck, build, native Runner, browser, and release verification gates. Greptile rates this head 5/5; all review threads are resolved. - Head `2626c8f9b` passed all 56 checks: 54 passed and two intentionally skipped. This includes full typecheck, build, native Runner, server tests, serialized suites, browser E2E, and Canary Dry Run. The unchanged Cursor managed-runtime test exceeded its five-second deadline on the first attempt; its focused local suite passed all eight cases, and the single CI rerun plus dependent verify gate passed. - A prior full local serial test attempt reported 19 failures (16,141 passed, 88 skipped) and stopped before later wrapper groups. It began before the final source edits. Every failed suite has a passing fresh recheck; the macOS snapshot stress case passes alone in 227 seconds. The previous head `000fb9261` passed all 56 CI checks. PRs leave `pnpm-lock.yaml` unchanged; CI and the refresh bot own dependency resolution. ## Risks - Dot remains off by default. Its own opt-in does not enable other Runner providers. It still requires the MCP option; disabling Dot blocks new work while keeping existing recovery and saved bindings. - Dot requires a stable public HTTPS origin. Its base provider in #15402 is merged. Temporary tunnels are useful for testing but are not permanent deployments. - Idle intake creates a visible agent-authored task. It does not fabricate a human message or bypass ordinary admission. - Workspace commands require Linux bubblewrap with a private PID namespace; deployment qualification is still needed. macOS file tools and artifact publishing remain available, but commands are disabled because sandbox-exec does not contain detached descendants. There is no unrestricted fallback. Command protection fails closed above 4,096 workspace directories. The historical macOS command walkthrough does not qualify the current Linux implementation. - Provider model choice, token usage, cost, native thread control, and global external stopping remain unavailable. Known Paperclip budget gates still apply. - The workspace bridge and attachment reader are separate opt-in settings. Reading assigned task files sends their contents to OpenAI. Revocation prevents future reads but cannot withdraw bytes already sent. Files are read on request; automatic inbound attachment staging remains disabled. - An existing event registration can retain earlier instructions that prohibit extra tasks. Dot asked for permission before a second intake after the final event-only bootstrap; that extra no-nudge continuation is not live-qualified under that earlier registration. The later attachment qualification submitted normally through the composer. The revised onboarding prompt describes the new idle entry point, and its protocol polling path is tested. - OpenAI event delivery can be delayed; a webhook acknowledgement is not proof that Dot has begun work. - A running Dot can retain an old plugin catalog after tool refresh. Refresh and actual tool exposure must be checked before using new top-level actions. - Hosted and remote controller modes are not qualified. ## Model Used OpenAI Codex, based on GPT-6. The exact deployment ID and context window size are not exposed in this session. Capabilities used: reasoning, repository editing, code execution, test inspection, and browser control. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites and all fresh failure rechecks pass; the earlier full attempt is disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-07 19:44:17 -05:00
DottaandPaperclip	f669194298	fix(runner): propagate configured environment to future turns (#15451) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runners start provider processes and enforce a separate tool environment policy. > - The server resolves task environment bindings at each run boundary. > - Fixed launch allowlists dropped custom variables after resolution. > - Warm process reuse and a blanket fingerprint exclusion also hid configuration changes. > - This pull request carries a bounded, server-selected list of task variable names through each process boundary. > - The benefit is that later turns and tool commands receive configured values while ambient host secrets stay excluded. ## Linked Issues or Issue Description **What happened?** A native Codex agent could not use configured task credentials. Adding the variables in Settings did not fix the next turn. The provider sanitizer, runner launch, Rust provider launch, and shell policy each used fixed allowlists. Effective config fingerprints also excluded all variables with the `PAPERCLIP_` prefix. **Expected behavior** Explicitly bound task variables must reach provider processes and tool commands. Added, changed, and removed values must take effect at the next run boundary. A running turn keeps its original configuration. **Steps to reproduce** 1. Start a native Codex task without a custom environment binding. 2. Add a fake `PAPERCLIP_PAGE_BUCKET` value and a fake Pages credential binding in Settings. 3. Continue the task and inspect the tool environment. 4. Before this fix, those values are absent even from a fresh runner launch. **Paperclip version or commit** Reproduced at `b31558064`. The fix is rebased on current master. **Deployment mode** Self-hosted server with a native process runner. Related changes: [the legacy Codex MCP environment fix](https://github.com/paperclipai/paperclip/pull/13321) and [ambient server-secret exclusion](https://github.com/paperclipai/paperclip/pull/12870). These affect different launch paths. This change preserves their credential boundaries. ## What Changed - Capture scoped task bindings after resolution, before managed provider credential injection. Mint and validate the names-only projection at native dispatch before host inheritance. Legacy adapters retain their previous environment limits. - Strip user-supplied projection markers from agent, environment, project, and routine config. - Carry selected values through the Codex, ACPX, OpenCode, runnerd, and Rust subprocess launch boundaries. - Add selected names to native and ACPX Codex shell include lists. Keep selected values out of command arguments, including selected bootstrap values. - Reject malformed projections, reserved authority and loader names, missing values, null bytes, and oversized input. - Replace a retained native process when projected values change. Keep unchanged processes reusable. - Fingerprint custom namespaced variables while excluding known generated runtime variables. - Document next-run behavior and add regression coverage. ## Verification - Red: the permanent reproduction failed at four launch/tool boundaries and the namespaced fingerprint check. Two control checks passed. - Green: the initial regression plus existing Codex environment and shell tests passed (39 tests). - Server config resolution, fingerprints, and native-session suites passed (653 tests), including addition, rotation, removal, and unchanged warm-session reuse. - Rust regression tests passed. A real shell child received added and rotated values, then lost them after removal. Unselected host variables stayed absent. - `pnpm -r typecheck` and `pnpm build` passed after rebase. The full `pnpm test:run` was attempted but could not complete: fresh embedded PostgreSQL databases fail during bootstrap on this macOS host. An isolated suite and a disposable native `initdb` probe reproduced the failure before test execution. `shmget` reports `No space left on device` because the host has exhausted shared-memory IDs. This is not disk exhaustion. The run was stopped after confirming the external setup failure. CI results will be recorded separately. - Runner boundary suites: 361 tests passed. Two process-launch errors during concurrent binary staging passed on isolated rerun. - Rust Codex provider and process supervisor integration suites: 97 passed, 2 intentionally ignored. - ACPX shell and selected-bootstrap argv regressions: 3 failed before the fix, then all 55 relevant tests passed. - Review compatibility regression: 129-variable and large-value legacy configurations failed before the correction and passed after moving native-only validation to dispatch. - Final review head `3f11e8d25`: repository typechecks and production build passed. CI completed its implementation checks; one general-server shard hit SQL `40P01` in `heartbeat-runtime-skills.test.ts` during its `beforeEach` table truncate (1,199 tests passed in that shard). The shard passed on its single rerun. All CI checks for this head are green. Greptile reviewed this head at 5/5 with no new actionable findings; the compatibility thread is resolved. - No live provider credentials or model calls are required by these tests. ## Risks - Configured task credentials now reach the tools they were configured for. The controller selects names only after existing scope and secret-binding authorization. - TypeScript and Rust validate the same bounded projection. Their reserved-name rules must stay aligned. - A changed projection replaces an idle provider process. Unchanged values preserve reuse. Active turns retain their original environment. - No database migration or API schema change. > This fixes existing runner configuration behavior. It does not add a new roadmap capability. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, local code execution, and repository tools. The session does not expose a more specific API model identifier or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-07 10:16:49 -05:00
DottaandPaperclip	1640c5b6ab	feat(ui): render HTML artifacts in a secure sandbox (#15447) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents deliver reports as task attachments and workspace files. > - The board already renders Markdown, but it shows HTML reports as source or rejects their preview. > - Reports can need inline scripts to build charts and tables. > - Artifact scripts must not read board cookies, storage, or the parent page. > - This pull request adds an opaque-origin HTML preview and view controls beside Download. > - Users can explore a report, inspect its source, and download the original file. ## Linked Issues or Issue Description **What existing behavior does this improve?** The task attachment panel and workspace file viewers. **Subsystem affected** ui/ and the server workspace file preview service. **Current behavior** HTML attachments show source text. Workspace HTML files cannot be previewed. **Proposed behavior** Render self-contained HTML reports in a sandboxed iframe. Place eye and code controls beside Download. Preserve the original file for raw view and download. **Reason and benefit** Users can read and filter agent reports in Paperclip without giving artifact scripts access to the board session. **Breaking changes** HTML previews now open rendered. Scripts and styles must be embedded in the report. Remote resources, API requests, forms, popups, and host navigation are blocked. Workspace HTML content uses the existing bounded UTF-8 JSON response. **Additional context** Related closed proposals: #3293 and #4857. This change uses the current attachment and workspace viewers and adds no report-serving endpoint. The Artifacts and Work Products roadmap item is complete; this improves its existing preview behavior. ## What Changed - Add a shared HTML iframe renderer with `sandbox="allow-scripts"` and no `allow-same-origin`. - Install a restrictive CSP before artifact markup. Keep inline report scripts and styles. - Add shared rendered/raw icon controls beside Download in the attachment panel, workspace panel, and file sheet. - Return workspace HTML as bounded text inside JSON. Keep download and path-access protections. - Add Storybooks for reports, security probes, workspace viewers, a task journey, and mobile layouts. - Document the security boundary and browser test command. ## Verification - `pnpm -r typecheck` and `pnpm build` pass. - Token gates and the static Storybook build pass. - Four browser tests pass. They cover report filters, raw mode, task entry, workspace viewers, cookies, storage, host DOM, resource requests, forms, popups, and host navigation. - The focused UI suite passes 26 tests. The file-resource server suite passes 36 tests. - Local full-stack acceptance passes with an actual 52 KB HTML report. Its chart and filter render. Raw mode preserves the source. Download bytes match the original file. - The complete local UI suite passes: 698 files and 7,754 tests. - `pnpm test:run` was attempted. Its server group recorded two unrelated timeouts and eight connector failures. All failed cases pass in isolated reruns, including the complete 388-test connector suite. The remaining local run was stopped after the full CI test coverage passed, to release test database resources. The full local command did not complete cleanly. - The separate local CLI group passes 511 tests. Three worktree database cases fail to start embedded PostgreSQL because this Mac has exhausted its shared-memory allocation. One isolated rerun fails at the same database startup step. No CLI code was changed. The corresponding CI test jobs pass. - All 56 PR checks are clean on `c771255f1a2e86bd8825f22eafb5d14a5e342cce`. Greptile gives 5/5 with no actionable findings. The security scan passes. The branch has no merge conflict. - Review the **HTML artifacts** section in Storybook. Run `pnpm exec playwright test --config tests/html-preview/playwright.config.ts` for the browser checks. ## Risks - Never add `allow-same-origin` to this iframe. Its opaque origin is the cookie, storage, and host-DOM boundary. - Reports that depend on CDN assets need to embed their dependencies. - The frame can navigate itself. Sandbox restrictions remain after navigation. This is not full network isolation. - Inline scripts can consume browser resources. This change does not isolate CPU or memory use. - No database migrations or API shape changes are required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository editing, code execution, and browser tool use. The runtime does not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass — feature and UI checks pass; the full local run has the infrastructure limits recorded above. All CI test jobs pass. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-07 09:01:05 -05:00