mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
ad5875fa4b78804e77911703f7ab2aacb9b32067
467
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
4ba521823f |
merge: preserve Claude consumers and canonical install diagnostics
Retain both Claude SDK admission and bounded Intel initial-install failure evidence. All25 focused consumer controls pass; no provider, build campaign, local Rust or Docker. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
12113e431e |
merge: reconcile task-workspace canonical prerequisite
Preserve Claude v7 effort, lease ownership, historical readers, and Pi25 source binding while composing canonical16a and task-workspace masterf329. Protocol/surfaces noemit and all generation checks pass. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
16a53254c4 |
merge master and retain bounded install failure evidence
Preserve guarded test-home cleanup and both Storybook source reviews. Keep the installer deadline and all artifact/lifecycle/runtime assertions intact while retaining its failed npm progress. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f29dad6c94 |
merge: preserve current canonical source and Claude fixture review
Keep the qualified four-target selection and both Linux provider-pack assembly paths while carrying the current shared source. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
48ffca9438 |
merge: reconcile published native prerequisite and current master
Preserve canonical startup ownership, scoped evidence, current Storybook IDs, and the Codex form controls while carrying Claude effort and immutable source bindings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c8f2c26577 |
chore: apply contribution rules to platform qualification
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c1b9933a24 |
chore: follow source comment contribution rules
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9f59ff36cf |
merge: reconcile runtime qualification with current master
Co-Authored-By: Paperclip <noreply@paperclip.ing> * origin/master: fix(server): keep classified provider quota runs out of Sentry (#16029) refactor(chat): extract delivery, reaction, and setup boundaries (#16024) fix(heartbeat): answer an unsettled adapter Stop with a 409 instead of a 500 (#16032) refactor(storybook): organize folders and navigation by feature map (#16049) test: remove repeated setup and strengthen regression proof (#16022) feat: maintain Storybook through a downstream feature catalog (#16046) ci: route PR admission through reserved AWS capacity (#16042) docs(release): canonicalize v2026.1012.0 release notes (#16045) ci: add protected routing workflow for reserved AWS capacity (#16039) docs: add PR rules for file size and code comments (#16036) Add managed Slack setup with an independent own-app path (#15679) |
||
|
Dotta
|
579192c560 |
Merge branch 'codex/claude-runner-default' into codex/claude-runner-platforms
* codex/claude-runner-default: docs: record Claude setup and attestation repair evidence fix(claude): use discovered runner in guided agent setup fix(runner): version Pi attestation for owned settings initialization |
||
|
DottaandPaperclip
|
0ff31b2b76 |
fix(runner): version Pi attestation for owned settings initialization
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
88bf4c6c1d |
refactor(storybook): organize folders and navigation by feature map (#16049)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The feature map describes product areas and verification entry points. > - Storybook has 223 story suites that show production components, proposals, and retained history. > - The parent PR maps those suites to features, but their folders and sidebar still use unrelated groups. > - This pull request puts the existing suites under feature folders and uses the feature map's names and order in the sidebar. > - Explicit meta IDs preserve existing story links while the titles change. > - Contributors can find stories through the same product structure without losing examples. ## Linked Issues or Issue Description Refs #16046. This is the requested follow-up. Parent #16046 is merged. This PR now targets master, with its single follow-up commit rebased onto the parent squash merge. The Storybook implementation is unchanged by that rebase. Related open Storybook PRs were checked; this change organizes existing suites and adds no product capability. ## What Changed - Move all 223 story files into folders named for their primary feature area. Keep useful scenario groups below the feature's sidebar label. - Order the sidebar by the eight feature-map sections. Put Overview first and shared Foundations last. Leave areas with no stories out of the sidebar. - Move 19 support files into `ui/storybook/support/` and update relative imports and active documentation. - Preserve all 2,210 story IDs and the single docs ID through explicit CSF meta IDs. Keep all entry names and story exports. - Record stable IDs and sidebar titles in the downstream catalog. Validate feature folders, title prefixes, and IDs. Update the maintenance skill and review evidence. ## Verification - Storybook build passed. Built index comparison against parent `75bd3c977cd828d5222823850339e8204f40eb47` found the same 2,211 entries, IDs, and names. - UI typecheck passed (`ui/node_modules/.bin/tsc -b` from `ui`, invoked as `./node_modules/.bin/tsc -b`). - UI Vitest project: 728 files and 8,176 tests passed. - Catalog checker and its nine tests passed. Feature-map checks, token gates, and the maintenance-skill validator passed. - Chromium: nine existing play cases and one render case reached `storyFinished: success` with no page exceptions. The guide rendered, sidebar sections followed map order, and a search for `steering/queue` opened its preserved story link. - Source comparison found only meta title/ID changes, rebased relative references, and updated fixture path text in story files. - To inspect: build Storybook, start `node scripts/serve-storybook-static.mjs --port 6136`, and open the Overview guide. Expand feature folders and use existing story links. - The complete repository suite was not repeated for this organization change. The parent records a backend timeout that passed in isolation, without a complete broad-suite pass. Visual comparisons remain blocked by the unpublished empty baseline. The separate fixture-server story was not run. ## Risks - Open branches that add stories under old paths will need to choose a feature folder and update the catalog. Cross-feature suites have one primary home and retain related capability references. - Sidebar paths change; story and docs IDs remain stable. Automated references to source paths may need the new location. - This is one requested follow-up with 258 changed files, mostly moves. It exceeds the usual review-size guideline because splitting it would leave the sidebar and folder convention partly migrated. - Browser verification samples ten stories, not all 2,210. Existing production/prototype classifications and review gaps remain unchanged. - Required CI gates and a 5/5 review passed on the original head. Fresh checks and review are running for the rebased head. Catalog validation and all nine checker tests passed after rebase; UI, map, skill, and checker contents are identical to the original reviewed head. ## Model Used - OpenAI Codex, GPT-6. The exact runtime model ID and context-window size are not exposed in this session. Used reasoning, repository inspection, code execution, and local browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9e721264e9 |
feat: maintain Storybook through a downstream feature catalog (#16046)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Storybook supports review of its production UI and proposed designs. > - Its 223 story files have no consistent link to product capabilities or record of component and mock provenance. > - Old compositions can drift while they still look like current product examples. > - This pull request adds two maintenance skills and a catalog that consumes the existing feature map. > - The first audit preserves all stories and fixes proven fixture drift. > - Contributors can now find examples by feature and see what still needs review. ## Linked Issues or Issue Description **What existing behavior does this improve?** Storybook maintenance, feature discovery, and the accuracy of isolated UI examples. **Current behavior** Stories are already centralized under `ui/storybook/`, but their sidebar has 51 top-level groups. Production components, local prototypes, and historical designs are not consistently distinguished. The old guide uses manual coverage claims. GitHub searches found no duplicate feature-map catalog change. **Proposed behavior** Keep existing story locations and IDs. Add a downstream catalog keyed to stable feature capabilities. Record product status, rendered subject, data boundary, and declared review dependencies separately. Use dedicated maintenance and review skills to reconcile drift and report evidence. **Reason and benefit** The catalog makes existing examples reusable without treating mock behavior or old designs as production acceptance. The feature map remains independent of Storybook. ## What Changed - Add `maintain-storybook` and `review-storybook` skills with contributor guidance. - Inventory all 223 story files. Generate a feature index and provide searchable navigation in the existing Overview story. Track direct subjects through relative imports and local wrappers, including the primitives demonstrated by foundation suites. - Add a checker for inventory, capability references, duplicate IDs, and declared-source drift, with eight regression tests. The parser test covers real nested CSF files, export filters, explicit IDs, and play counts. - Preserve historical designs and classify mixed wrappers. Record the initial audit and remaining parity work. - Align the question-dismiss/send fixture with production behavior. Add dismissal and confirmation-reopen regressions and repair the editor selector. - Exclude three exported fast-response fixtures from story discovery and add the missing render-time callback spy. ## Verification - Passed: `node scripts/storybook-catalog.mjs`, `node --test scripts/storybook-catalog.test.mjs` (8 tests), and `node scripts/check-maps.mjs`. - Passed: validators for both new skills, `pnpm check:token-gates`, `pnpm -r typecheck`, and `pnpm build`. - Passed: final Storybook build; nine selected browser `play` cases; three fast-response render cases; feature-guide search/link navigation; question-fixture reset on reload. - Built-index comparison preserved all 2,208 original story IDs and added only two new regression cases. - Passed: `pnpm exec vitest run --project @paperclipai/ui` (728 files, 8,176 tests). - The full `pnpm test:run` has reported a 15-second timeout in the untouched backend case `connects advertised Workspace gmail.read without mutating definitions in authenticated`. The exact test passed standalone without changing code, expectations, or timeouts. The remaining broad run was stopped after this failure. The full suite is not verified. - Visual comparison is blocked by the existing empty baseline manifest. The managed-accounts API/DB fixture was not started. See `maps/storybook-map/REVIEW.md` for exact scope. ## Risks - This is an initial inventory and targeted audit. 188 suites still require deeper state and fixture parity review. - Review hashes cover declared dependencies. They do not follow all transitive production imports or prove semantic correctness. - The catalog requires upkeep when stories, mapped recipes, or declared subjects change. Shared mocks still need feature-by-feature contract review. - No production application source, database schema, or upstream feature-map contract changes. ## Model Used OpenAI GPT-6 via Codex. Used reasoning, shell tools, file editing, and Chromium browser verification. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e7d4eb75a2 |
merge: carry current import preservation into platform qualification
Co-Authored-By: Paperclip <noreply@paperclip.ing> * codex/claude-runner-default: fix(claude): preserve permission selections and explicit import removals fix(runner): report the validated Pi profile in install evidence docs: record Codex configuration review and restart proof fix(agents): preserve requested controls and imported native settings fix(runner): preserve native Codex reasoning effort |
||
|
DottaandPaperclip
|
66f20c4682 |
fix(runner): report the validated Pi profile in install evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
fa064d247d |
merge: reconcile measured sandbox qualification with Claude platforms
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
e23dd02651 |
Merge remote-tracking branch 'origin/codex/codex-runner-default' into codex/claude-runner-protocol
* origin/codex/codex-runner-default: docs: record final environment metadata review closure fix(environments): preserve measured proof atomically on metadata saves test(agents): distinguish measured and unknown sandbox defaults test(e2e): qualify managed Codex images in existing fixtures fix(agents): preserve legacy policy and require measured sandbox defaults fix(environments): bind platform facts to the effective immutable image fix(runner): close review gaps in setup and platform qualification |
||
|
DottaandPaperclip
|
f5346c05b9 |
ci: route PR admission through reserved AWS capacity (#16042)
## Thinking Path
> - Paperclip uses CI to verify changes before release.
> - Trusted PR tests can run on isolated AWS workers.
> - Their runner-selection job currently waits for GitHub-hosted
capacity.
> - A large hosted test queue therefore blocks admission to AWS.
> - A dedicated routing workflow lets us reserve capacity for metadata
validation.
> - This PR moves the routing job to reserved AWS capacity after the
bootstrap workflow and pool are available.
## Linked Issues or Issue Description
**Issue type**
CI reliability enhancement.
**Where is the issue?**
The Select trusted runner job in PR CI.
**What's wrong?**
The routing job competes with a large hosted test queue. Its sampled
median wait was about 20 minutes. AWS tests cannot begin before this job
finishes.
**Suggested fix**
Add a protected reusable routing workflow. Deploy its dedicated AWS
pool, then switch the PR caller in a follow-up PR. No overlapping open
PR or issue was found.
## What Changed
- Call ci-routing.yml on protected master from pr-trusted.yml.
- Check the canonical repository and approved PR author/sender before
allocating reserved AWS metadata capacity. Unapproved PRs continue to
use hosted routing.
- Permit approved manual routing canaries on protected master.
- Use a validated roster snapshot for placement. Invalid JSON, missing
settings, stale snapshots, and unapproved identities select hosted
routing without a placement-time JSON parse.
- Preserve the full in-job identity, rerun actor, repository, head,
merge, and stack checks.
- Update stack-scope and placement regression tests and document the
pool and rollback.
## Verification
- The routing scripts and placement expression tests passed (44 tests),
routing-plus-shard tests passed (56 tests), and the existing
.github/scripts suite passed (381 tests).
- Existing routing allow/deny scenarios passed with the extracted script
(16 cases).
- actionlint passed with the dedicated custom runner label declared in
its local configuration.
- git diff --check passed.
- Full pnpm typecheck/test/build cannot run in this worktree because
dependencies are not installed. The change is limited to workflow YAML
and Node tests.
- The dedicated pool is deployed with two warm On-Demand workers and a
four-worker ceiling. The routing qualification passed: it started about
five seconds after workflow creation, selected AWS, and finished while
the hosted gate in the same PR remained queued.
- Bootstrap https://github.com/paperclipai/paperclip/pull/16039 is
merged. The deployed group now authorizes only ci-routing.yml on
protected master, and its master canary passed.
## Risks
- The maintainer explicitly approved a queued-CI admin bypass after
current-head Greptile 5/5 and focused verification.
- Activation requires the deployed routing Fleet and its group to
authorize the routing workflow on protected master. Keep the bootstrap
dependency explicit.
- The group must authorize only ci-routing.yml on protected master and
canonical repository ID 1170821064.
- The routing workflow runs no checkout or PR code and has read-only
permissions. The test admission checks are unchanged.
## Model Used
- OpenAI GPT-6 via Codex, with reasoning, shell tools, and file editing.
The exact runtime model ID and context window are not exposed in this
session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused routing and
workflow tests; full pnpm commands are blocked by missing worktree
dependencies)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required CI status checked; the maintainer explicitly approved
admin bypass for queued CI after clean current-head review and focused
verification
- [x] Greptile is 5/5 on
|
||
|
DottaandPaperclip
|
2a19f3721d |
fix(runner): close review gaps in setup and platform qualification
Read inert exact-source target data before selecting hosted native jobs. Bound registration teardown and retain unconfirmed recovery state. Record company-scoped sandbox platform facts only for immutable OCI images after confirmed cleanup; preserve existing connection and runner behavior. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
02ad4267d4 |
Merge branch 'codex/claude-runner-default' into codex/claude-runner-platforms
* codex/claude-runner-default: fix(adapter-utils): preserve SSH command positional arguments feat(runner): qualify native Codex platforms and cache hosted builds docs: establish paired feature and agent access maps (#16025) fix: retain matching-host runner development overrides fix(acpx): record turn activity at the moment a stop fires (#15962) refactor(chat): extract channel status, receipts, drafts and authority workflows (#15995) fix: preserve saved setup and match sandbox runner targets feat: add portable verification skills and refresh the feature map (#15983) |
||
|
Dotta
|
7eb4f8bdba |
Merge remote-tracking branch 'origin/codex/codex-runner-default' into codex/claude-runner-protocol
* origin/codex/codex-runner-default: feat(runner): qualify native Codex platforms and cache hosted builds docs: establish paired feature and agent access maps (#16025) fix: retain matching-host runner development overrides fix(acpx): record turn activity at the moment a stop fires (#15962) refactor(chat): extract channel status, receipts, drafts and authority workflows (#15995) fix: preserve saved setup and match sandbox runner targets feat: add portable verification skills and refresh the feature map (#15983) |
||
|
DottaandPaperclip
|
ddcd746eb6 |
feat(runner): qualify native Codex platforms and cache hosted builds
Extract the tested runtime and installed-package prerequisite without changing agent creation defaults or experimental gates. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7d9b93aee2 |
docs: establish paired feature and agent access maps (#16025)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors use the feature map to understand and verify product capabilities. > - Agents need APIs and usable entry points within their permission boundaries. > - Product recipes and agent access need separate records with shared capability IDs. > - This pull request establishes that contract and gives maintenance ownership of both maps. > - The benefit is repeatable access review without changing product behavior or permissions. ## Linked Issues or Issue Description Refs: #15983, which added the portable verification skills and updated the feature map. This is its contract-only follow-up. Related: #15288 added the original feature map. **Issue type** Missing documentation and maintenance contract. **Where is the issue?** The product map, its missing agent access companions, and the verification skills. **What's wrong?** There is no shared convention for API presence, agent entry points, permission boundaries, access gaps, or evidence. An API declaration does not prove that a permitted native or remote agent can use it. **Suggested fix** Keep the product and agent access maps as siblings under `maps/`. Reuse capability IDs. Record representative source inventory separately from operation review and live proof. Keep both maps aligned through the maintenance skill. ## What Changed - Move the existing product map to `maps/feature-map/`. Preserve its recipes and Poteto format. Update relative links and add companion links. - Add `maps/agent-access-map/` with 35 area files and 171 matching capability IDs. Seed representative API operations, agent paths, policies and known gaps from the reviewed source inventory. Mark entries as `baseline`, `source-only`, and live verification `not-run`. - Define the shared contract in `maps/README.md`. Track API presence, agent availability, policy, evidence and review depth separately. Require operation enumeration before marking a broad capability operation-reviewed. - Extend `maintain-verification-skill` to reconcile both maps and inspect actual route policy and production tool admission. Update creation, verification, notices and contributor references to use the shared contract. - Add `scripts/check-maps.mjs` and ten regression tests. Check matching areas/IDs, indexes, references, contract fields, API method/path examples, justified exceptions and evidence-state consistency. - Keep the coverage inventory unchanged. Preserve skill invocation policies and both MIT license blocks. No product API, permission, UI, runtime helper, migration, dependency or workflow changes. ## Verification - Passed: `node scripts/check-maps.mjs` reports 35 paired areas and 171 matching capability IDs. All local map references resolve. - Passed: `node --test scripts/check-maps.test.mjs` runs ten tests. Fixtures cover missing companions, renamed/orphan/duplicate IDs, broken references, missing operation fields, empty exceptions, missing API examples, empty policy fields, and verified evidence without passed verification. - Passed: metadata validation for all three verification skills. Skill and contributor links resolve. Invocation policies and both license blocks are unchanged. - Compared relocated recipes with current master. Contents differ only in relative links and companion links. `coverage.json` is byte-for-byte unchanged. Master's updated read-only interaction test reference is retained. - Passed: `git diff --check`. The branch starts from current master and contains only this contract change. - Repository-wide typecheck, tests and build were not run locally for this documentation and Node structural-check scope. Live agent access was not exercised. Those gaps are explicit in the baseline entries; the structural check does not certify behavior. CI and automated review are pending. ## Risks - The companions are representative source baselines. They are not complete operation inventories or passed agent journeys. - The checker cannot establish actual permissions, native tool admission, operation completeness or truthful live evidence. Maintenance must inspect source and verify with the actual permitted agent context. - External links to the old product-map directory may need updates. Repository references have been updated. - Existing access gaps and board/user restrictions remain recorded. This contract does not change authority or implement missing paths. ## Model Used OpenAI GPT-6 through Codex assisted with this change. The session does not expose the exact serving model ID, configured context-window size, or reasoning setting. Capabilities used: source analysis, file edits, shell commands, and structural regression tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0953983526 |
Verify installed Claude SDK admission in native clean consumers
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
33077bed73 |
Pin provider-pack producer Node to the image version
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
adf258ede2 |
Merge latest Codex qualification and master fixes
Co-Authored-By: Paperclip <noreply@paperclip.ing> * codex/claude-runner-default: (25 commits) fix: pin ARM qualification helper to workflow source perf(ci): cache the Runner's release crate artifacts keyed by their inputs (#15932) perf(build): reuse a current Runner build in the server's prepare:runner-vendor step (#15926) fix(test): make the vitest home teardown remove read-only runtime-context bundles (#15931) fix(test): stop the Sentry timeout assertion matching a random event id (#15929) test: qualify Linux ARM with stock sandbox prerequisites docs(release): align 2026.1006.0-beta.0 stable notes header with v2026.1012.0 (#16000) fix: use Hermes adapter API URL env in prompt (#9425) fix(docker): skip the full-tree ownership walk on routine boots (#15992) refactor(chat): isolate Telegram, Slack, GitHub and Discord workflows (#15976) Preserve repository dependencies when importing GitHub skills (#15731) refactor(heartbeat): extract instruction working-copy lifecycle (#15984) test: retire redundant suites and consolidate owner coverage (#15978) refactor(heartbeat): extract runtime service coordination (#15980) refactor(heartbeat): extract execution configuration (#15973) refactor(heartbeat): extract run output recording (#15969) refactor(chat): extract failed-run retry coordination (#15845) refactor(chat): extract publication replacement decisions (#15844) refactor(chat): extract identity linking (#15836) refactor(heartbeat): extract session preparation (#15966) ... # Conflicts: # packages/paperclip-runner/package.json |
||
|
Dotta
|
12f7b62115 |
Merge latest Codex qualification prerequisites and master fixes
Co-Authored-By: Paperclip <noreply@paperclip.ing> * origin/codex/codex-runner-default: (25 commits) fix: pin ARM qualification helper to workflow source perf(ci): cache the Runner's release crate artifacts keyed by their inputs (#15932) perf(build): reuse a current Runner build in the server's prepare:runner-vendor step (#15926) fix(test): make the vitest home teardown remove read-only runtime-context bundles (#15931) fix(test): stop the Sentry timeout assertion matching a random event id (#15929) test: qualify Linux ARM with stock sandbox prerequisites docs(release): align 2026.1006.0-beta.0 stable notes header with v2026.1012.0 (#16000) fix: use Hermes adapter API URL env in prompt (#9425) fix(docker): skip the full-tree ownership walk on routine boots (#15992) refactor(chat): isolate Telegram, Slack, GitHub and Discord workflows (#15976) Preserve repository dependencies when importing GitHub skills (#15731) refactor(heartbeat): extract instruction working-copy lifecycle (#15984) test: retire redundant suites and consolidate owner coverage (#15978) refactor(heartbeat): extract runtime service coordination (#15980) refactor(heartbeat): extract execution configuration (#15973) refactor(heartbeat): extract run output recording (#15969) refactor(chat): extract failed-run retry coordination (#15845) refactor(chat): extract publication replacement decisions (#15844) refactor(chat): extract identity linking (#15836) refactor(heartbeat): extract session preparation (#15966) ... # Conflicts: # tests/runner-e2e/pi-controls-evidence.test.ts |
||
|
DottaandPaperclip
|
665c155472 |
Merge master and preserve qualified runner paths
Retain both hosted cache scopes. Use shared test-home teardown and restore owned directory permissions for the observed Mac ENOTEMPTY failure while keeping unrelated errors visible. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a5b16834c8 |
fix: pin ARM qualification helper to workflow source
Keep frozen product and reusable workflow tooling identities separate. Preserve clean-consumer and package integrity checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
8227485372 |
perf(ci): cache the Runner's release crate artifacts keyed by their inputs (#15932)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Its trusted PR workflow builds the Runner's Rust binary in five
jobs: Typecheck + Release Registry, Build, Canary Dry Run, and the
`rust` and `vitest 2/2` lanes of Verify Paperclip Runner. The `Build`
job is the slowest check in almost every green run (461s, 489s and 515s
in runs 38180574768, 38182717052 and 38183149920 on 2026-10-11).
> - Each of those jobs restores master's `release-runner-v2` dependency
cache. That cache holds only third-party crates. `rust-cache` removes
workspace crates when it saves, and a fresh checkout invalidates cargo's
mtime fingerprints for workspace crates in any case. So cargo compiles
`paperclip-runner-core` in release mode from scratch on every run. The
job logs show one `Compiling` line and 28s to 49s for it.
> - 18 of the 239 commits on `master` in the seven days before this
change touched the crate's inputs. The other 221 compiled the same
sources to the same binary again.
> - This pull request adds a second cache that holds only that crate's
release artifacts (about 70 MB). The key is made from the compiler, the
checkout root and a hash of every input the build reads. The master Rust
lane writes it. A pull request whose merge tree has byte-identical
inputs restores it and marks the artifacts newer than the checkout, so
cargo keeps them. Every other pull request misses and compiles as
before.
> - The benefit is about 28s to 49s less on `Build`, which is the run's
critical path, and the same on Typecheck, Canary Dry Run and two Runner
lanes, for about nine of ten pull requests.
## Linked Issues or Issue Description
No GitHub issue exists for this. The description below follows the
enhancement template.
**What existing behavior does this improve?**
The time the trusted PR workflow spends to compile the Runner's own
crate on every run.
**Subsystem affected**
CI / build
**Current behavior**
Every PR job that builds the Runner binary compiles
`paperclip-runner-core` in release mode from scratch. The compile takes
28s to 49s on the `runs-on` fleet (one `Compiling paperclip-runner-core`
line, then `Finished release profile` in runs 38169436532 to 38183149920
on 2026-10-11). The shared Rust cache cannot help: it holds only
third-party crates, and a fresh checkout gives every source a new mtime,
which makes cargo discard the crate's fingerprints.
**Proposed behavior**
The master Rust lane in `release-verify.yml` saves the crate's release
artifacts under a key made from the compiler, the checkout root and a
hash of every input the release build reads. The PR jobs that build the
binary restore that entry read-only. On a hit, a script sets the mtime
of every restored artifact to now, so cargo sees every source as older
than the build and keeps the artifacts. `pnpm build` then prints
`Finished release profile` in under a second for that crate. On a miss,
cargo compiles the crate exactly as before.
**Reason and benefit**
About 28s to 49s off `Build`, the slowest check in almost every green
run, and the same off the Typecheck, Canary Dry Run and two Verify
Paperclip Runner lanes. The saving applies to every pull request whose
merge tree does not change the crate's inputs, which was 221 of 239
`master` commits in the week before this change.
**Breaking changes**
None. The checks are unchanged. A miss or an eviction costs the compile
time it costs today and nothing else.
**Additional context**
This is a workflow change. `pr.yml` calls `pr-trusted.yml@master`, so
this pull request's own run uses the current reader steps, and the
writer runs only on a master push. The saving becomes visible on the
first PR runs after the merge and after the next master push has written
the entry.
## What Changed
- `scripts/runner-release-crate-cache.mjs` (new): the one place that
defines the cached path list (`CACHED_PATHS`), the hashed inputs
(`HASHED_INPUTS`) and the files a restore must contain (`REQUIRED`). The
`scope` subcommand prints the platform, the `rustc --version --verbose`
release, commit and host, and a hash of the checkout root. The
`mark-current` subcommand sets the mtime of every restored artifact to
now and prints a `paperclip.runner-release-crate.v1` JSON line with the
file count and bytes. If the restored set lacks any of the seven files
cargo's freshness check reads or checks for (the fingerprint hash and
dep-info of each unit, the lib's rlib and rmeta, the binary under
`deps`), it removes the restored files, prints `action: discarded`
naming the missing file, and exits 0, so cargo compiles as on a miss.
- `.github/workflows/pr-trusted.yml`: after the `release-runner-v2`
restore in `typecheck_release_registry`, `build` and `canary_dry_run`,
and in the `rust` and `vitest 2/2` lanes of `verify_paperclip_runner`
(new matrix key `release_crate: true`), three steps: compute the key
scope, `actions/cache/restore` the crate under
`runner-release-crate-v1-<scope>-<hashFiles(...)>` with no
`restore-keys`, and run `mark-current` when `cache-hit == 'true'`. The
`hashFiles` list covers the toolchain pin, the manifests and lockfile,
the crate sources, the protocol tree and the three fixture trees the
test modules include (two under the Runner package and
`tests/runner-e2e/fixtures` at the repository root).
- `.github/workflows/release-verify.yml`: in the Rust lane of an
own-repository master push, after the debug warm-up: compute the key
scope, look the key up (`lookup-only: true`), and on a miss run `cargo
clean --release -p paperclip-runner-core` and `pnpm build:binary` (the
exact command the readers run, so the entry carries no test-profile
variants), then `actions/cache/save` the same path list under the same
key.
- `scripts/__tests__/runner-release-crate-cache.test.mjs` (new, 10
tests, in `test:release-registry`): the path list and hashed inputs are
the expected ones and exist; every `include_str!` and `include_bytes!`
in the crate sources resolves under a hashed input, whether its argument
is a literal or a `concat!` of literals, `env!("CARGO_MANIFEST_DIR")`
and macro metavariables, and any other form fails the test by name (this
guard found the `test/fixtures` tree during development and the
`tests/runner-e2e/fixtures` tree in review); the parser itself is tested
on a synthetic crate for each supported form and six rejected ones;
every PR reader restores after the dependency cache and marks before its
build command with the script's exact path list and key and no
`restore-keys`; the Verify lanes are gated by `release_crate`; the
writer runs after the checks with the lookup, the clean and build, and
the save under the readers' exact path list and key; the scope changes
with the compiler and the checkout root and survives a missing `rustc`;
`mark-current` makes every artifact newer than the checkout, leaves a
dependency's artifacts alone, discards a restore missing any one of the
seven required files (one case each, checking the named reason), and
still marks a restore missing any one of the seven files cargo does not
need.
## Review fixes
Greptile's two findings on the first commit were both valid and are
fixed in the second commit:
- **Incomplete caches looked usable.** `REQUIRED` checked four files,
but cargo's freshness check needs seven. Measured with cargo 1.97.1 on
the real crate (restore, touch the artifacts, delete one file, build):
without the lib's fingerprint hash, dep-info, rlib or rmeta it
recompiles the crate (24s to 27s); without the bin's fingerprint hash,
dep-info or the binary under `deps` it relinks (1.5s); without either
`.json` fingerprint record, either rustc `.d` file, the uplifted binary
or its `.d` it reports the crate fresh in 0.15s and recreates the
uplifted binary from `deps`, and the staging step then copies it.
`REQUIRED` now lists exactly the seven files, the glob for the binary
under `deps` excludes the `.d` beside it, and the test removes each of
the seven in turn (discarded, naming it) and each of the other seven in
turn (still marked).
- **Composed include paths escaped the guard.** The regex matched only
`include_str!("literal")` and skipped `include_str!(concat!(...))`,
which the crate uses four times (30 of 34 includes were checked). The
guard now parses every include argument and resolves the `concat!`
forms; one of them, in `provider_events.rs`, reads
`tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json`
at the repository root, outside every hashed input. It is test-only, so
today's release binary does not read it, but the key is meant to cover
every include whatever its build, so `tests/runner-e2e/fixtures/**` (9
files, 80 KB) joins `HASHED_INPUTS` and the six key embeddings in the
two workflows. With the first commit's input list, the new guard fails
on exactly that include.
- `doc/RELEASE-AUTOMATION-SETUP.md`: a "Runner release crate cache"
section under the Runner verification dependency cache, with the key,
the writer, the readers and the trust boundary.
- `package.json`: the new test file in `test:release-registry`.
## Verification
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`:
10 tests pass, three consecutive runs.
- The second commit, with the real toolchain on the real crate (`rustc
1.97.1`, cold build 2m20s, the crate's artifacts 74 MB): every source
touched to a fresh-checkout mtime and the archived artifacts restored,
cargo compiles (`Compiling paperclip-runner-core`, 24.8s and 25.1s, the
two controls); restored and marked by `mark-current`, `Finished release
profile in 0.16s` with no `Compiling`, and the same on a second build.
Then, for each of the fourteen files cargo wrote for the two units, one
at a time: restored, marked and that file deleted, cargo recompiles
without any of the seven `REQUIRED` files and reports fresh without any
of the other seven; restored with that file deleted and run through
`mark-current`, the seven required ones print `action: discarded` naming
the file and the others print `marked`. A restore holding only the seven
required files builds fresh in 0.15s, cargo recreates the uplifted
binary, and `stage-runner-binary.mjs` copies it into `dist/bin`.
- `node --test '.github/scripts/tests/*.test.mjs'`: 378 of 381 pass. The
3 failures are in `docker-canary-promotion.test.mjs` and fail the same
way on an unmodified `master` checkout in this sandbox (the stand-in
docker writes no `calls.jsonl`).
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs`: 99 of 101 pass; the
2 failures spawn `pnpm`, which is not on this sandbox's PATH, and fail
the same way on `master`.
- Both workflows parse as YAML. No job has a duplicate step name or step
id.
- The mechanism, with the real toolchain (`rustc 1.97.1`, the package's
pin) on a copy of the crate: a cold `cargo build --release -p
paperclip-runner-core --bin paperclip-runnerd` (2m23s here). Then the
artifact set was archived with `tar`, removed, every source re-copied so
each had a new mtime, and the set restored:
- restored, not marked: `Compiling paperclip-runner-core`, 24.6s (the
control; this is what CI does today);
- restored and marked by `mark-current`: `Finished release profile in
0.15s`, no `Compiling`; a second build is the same;
- one Rust source edited after a marked restore: `Compiling`, 24.0s;
- one protocol schema edited after a marked restore: `Compiling`, 23.9s
(the protocol files enter cargo's dep-info as absolute paths and are
hashed into the key);
- the rlib removed from a restored set: `mark-current` prints `action:
discarded, reason: incomplete: the lib rlib is missing` and removes the
rest, and cargo compiles.
- Not run here: the workflow steps themselves. `pr.yml` pins
`pr-trusted.yml@master`, so this pull request's own `Build` job runs the
current reader steps, and the writer runs only on a master push. The
first PR runs after the merge and after the next master push show the
`Restore the Runner release crate` step's result and the
`paperclip.runner-release-crate.v1` line in their `Build` logs.
## Risks
- Correctness: a reader can only restore artifacts that master built
from inputs byte-identical to its own merge tree, because every input
the build reads is in the key, and the key has no prefix match. A pull
request cannot write. The trust boundary is unchanged: master writes, a
pull request restores master's entry.
- An input the build reads that is outside the hashed patterns would let
a stale artifact pass. The release dep-info lists the crate sources and
the protocol schemas and fixtures; all are hashed. The guard test
resolves every `include_str!` and `include_bytes!` in the crate, literal
or `concat!`, and fails if one lands outside the hashed inputs or uses a
form it cannot resolve, so a new include form cannot slip past it by not
matching a regex.
- GitHub-hosted runners check out under `/home/runner/work`, the
`runs-on` fleets under `/home/runner/_work`. Cargo records the included
protocol files as absolute paths, so an entry from one root is useless
under the other. The checkout root is in the key, so the other root
misses instead of restoring something cargo then discards.
- A cache that does not keep mtimes on restore changes nothing: the
artifacts are then already newer than the checkout, and `mark-current`
sets them to now in any case.
- Cost: one new entry of about 70 MB per distinct input set, written
only by the master Rust lane on a miss, plus one `cargo clean` and
`build:binary` (about 40s to 50s) on that lane when the inputs changed.
The readers pay one restore of about 70 MB (two to three seconds at the
fleet's observed restore speed) and a sub-second `mark-current`.
- An incomplete entry (a writer that saved a partial set) is discarded
by `mark-current` and costs one compile, the same as a miss.
## Model Used
- Claude (Anthropic), model id `claude-fable-5-1` (Claude Fable 5.1),
extended thinking, with tool use (shell, file editing, GitHub CLI, a
local Rust toolchain for the experiment) in a Paperclip agent session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
0690801f13 |
test: qualify Linux ARM with stock sandbox prerequisites
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
39fec166b9 |
Merge the exact Pi v23 install-probe digest
Co-Authored-By: Paperclip <noreply@paperclip.ing> * codex/claude-runner-default: fix(runner): pin the Pi v23 digest in its install probe |
||
|
DottaandPaperclip
|
33c9216614 |
fix(runner): pin the Pi v23 digest in its install probe
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a1ac0a7324 |
Merge the current Pi clean-install assertion
Co-Authored-By: Paperclip <noreply@paperclip.ing> * codex/claude-runner-default: fix(runner): verify the current Pi profile in clean installs |
||
|
DottaandPaperclip
|
ca52d2998c |
fix(runner): verify the current Pi profile in clean installs
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d0da37c221 |
feat(runner): assemble target-owned Claude Linux provider packs
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
4f399cec9f |
fix(docker): skip the full-tree ownership walk on routine boots (#15992)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Docker image keeps all instance state under `PAPERCLIP_HOME`,
and deployments mount a volume there
> - Before it drops privileges, the entrypoint makes sure that the
`node` user owns the home (#10530)
> - The ownership probe is a first-mismatch `find` over the whole tree.
When ownership is correct, `-quit` never fires, so every boot stats
every file on the volume
> - On homes with many agent worktrees and `node_modules` trees, this
walk takes minutes before Node starts. That is close to common 300s
healthcheck windows
> - This pull request keeps a cheap shallow probe on every boot. It runs
the full-tree probe only once per volume and node UID/GID, and records
the result in a marker file
> - The benefit is that routine boots no longer scale with the file
count, and the original root-owned-volume fix stays in place
## Linked Issues or Issue Description
Refs #10431 (the open follow-up about the remaining cost of the
ownership walk). Related: #10530 (the change that added the full-tree
probe).
**What happened?**
On a hosted multi-instance deployment, we measured the time from
container start to the first app log line. Instances with many files on
the volume (agent worktrees with `node_modules`) took about 190-235s.
Small instances took 5-22s. One 30 GB volume with few files took 24s, so
the cost follows the file count, not the volume size. Almost all of this
time is the `find "$home_dir" \( ! -user node -o ! -group node \) -print
-quit` walk in `scripts/docker-entrypoint.sh`. When ownership is already
correct, `-quit` never fires, so the walk stats every inode on every
boot.
**Expected behavior**
When ownership is already correct, the entrypoint pre-start phase is
short and does not depend on the number of files in `PAPERCLIP_HOME`. A
fresh root-owned volume, a UID/GID remap, and an upgrade from an older
image are still repaired before the privilege drop.
**Steps to reproduce**
1. Run the image with a volume on `/paperclip` that holds a few hundred
thousand files, all owned by `node:node`.
2. Restart the container.
3. Compare the container start time with the first server log line. The
gap grows with the file count, even though no `chown` runs.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
598e190cb3 |
test: bind installed native payloads and await saved state
Isolate actual Codex resolver fixtures from global bridge lookup and preserve host qualification blockers with literal evidence. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
123a188374 |
fix: separate Codex login and ARM host qualification
Keep daemon ABI admission separate from CLI login and preserve restricted Docker installation checks before native host startup. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
695506ad39 |
perf: cache native builds and verify Linux runtime targets
Preserve actual target selection, record the GNU libc baseline, and retain primary installed-startup failures for ARM qualification. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dcdf09a0f3 |
feat: include Linux ARM64 in native Codex runner targets
Ship a source-bound ARM64 daemon alongside Linux x64 and both Mac targets. Use native hosted builds and the existing clean npm/native startup verifier. Read target requirements from the immutable source to preserve historical three-target promotions. Keep Codex payloads as official npm dependencies. Resolve ARM64 creation and login through the shared platform contract. Share the existing legacy CLI permission calculation with import previews without changing execution behavior. Record targeted checks and live Mac evidence. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
058af8f1b6 |
fix: preserve release artifacts during dev builds and display imported policy
Keep source-built daemons separate from source-bound release targets and select fresh compatible source output only inside the development supervisor. Preserve release manifests, running executable inodes, and explicit operator overrides. Show the existing translated Codex permission policy in import controls while preserving runner-only override payloads. Record real Apple Silicon task and paused import proof alongside exact qualification sources. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d522822a5d |
fix: preserve imported runner choices and qualify Mac install fixtures
Forward explicit import runner overrides, preserve native harness identity without experimental Codex gating, refresh existing host daemon outputs atomically, and retain strict public install probes with target paths and the executing Node SDK. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f4abdc2cfc |
Merge master b189f586d into Codex runner default candidate
Preserve master transport, lifecycle, and release verification controls alongside macOS qualification. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7b8d29a856 |
feat: qualify macOS Codex runner defaults and release artifacts
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
db2f00947b |
fix(runner): include Claude ACP in the server package (#15912)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server ships a copy of Paperclip Runner for native agent execution. > - The Claude ACP bridge is a runner dependency, but it was absent from the server package. > - The runner resolves this bridge dynamically, so the import-based packaging check missed it. > - A Claude session then failed before startup and showed a generic connection error. > - This pull request ships the qualified bridge and reports a missing runtime with repair steps. ## Linked Issues or Issue Description **What happened?** A server install with the vendored runner could not load `@agentclientprotocol/claude-agent-acp/package.json`. The runtime test suggested checking the model or provider connection. The task reported an interrupted runner session. **Expected behavior** The server package includes its qualified Claude ACP runtime. A missing bundled runtime reports an installation error and explains how to repair it. **Steps to reproduce** 1. Use a server install with the vendored runner and no Claude ACP bridge. 2. Select the Paperclip Runner ACP harness and the Claude provider. 3. Run the runtime test or open a session. 4. Observe a missing-module failure before Claude starts. **Paperclip version or commit** Reproduced on `e733bddc45`. This branch is based on current master. **Deployment mode** Self-hosted server built from source. Related: #15224 upgrades the Claude bridge. This PR fixes its omission from the server package and keeps the current qualified version. A future bridge upgrade must update the server pin too. ## What Changed - Add Claude ACP `0.73.0` to the server dependencies and bundled dependencies. - Pin the qualified Claude SDK and bridge dependencies in the npm bundle producer graph. Publish its native packages as consumer optional dependencies instead of shipping the producer executable. - Bind source-built vendored runners to the shared pnpm dependency store while retaining the server manifest as the provider authority. Keep published npm and self-rooted deployment roots unchanged. - Include dynamically resolved Claude and Codex bridges in the vendor dependency check. - Preserve the missing-package category through the sidecar, Rust transport, and native session executor. Stop automatic retries for this installation failure. - Show **Runtime missing** in the runtime test and task thread. Explain the reinstall or rebuild, server restart, and runtime test steps. - Add regression tests and document the repair procedure. ## Verification - Red/green comparison with the production sidecar: the same missing-package setup returns a raw require-stack error on the original code and an actionable, non-retryable missing-runtime error on this branch. - The rebuilt server's vendored Claude installation probe passes using the source-server authority selected by the launcher, without starting a provider or making a model request. - Focused server verification: 830 tests passed. Runner installation and sidecar error tests: 83 passed, with 6 platform-specific skips. Packaging tests: 28 passed. Rust transport tests: 7 passed. - Source/deployment authority tests: 3 passed. Red/green packaging tests include a Linux-produced Claude SDK bundle, actual npm pack, and closed executable admission on Linux x64 and macOS ARM64/x64. - Full UI suite: 8,113 tests passed. - `pnpm -r typecheck` and `pnpm build` passed on the rebased branch. - Full current-head CI passed, including all general and serialized test lanes, Runner Rust/Vitest verification, typecheck, build, and all eight e2e shards. Greptile reviewed `d6a2741834ff922956560f2f689382827bbaf9e0` at 5/5 with no unresolved threads. - Local full-suite limitation: an earlier parallel attempt hit Postgres startup failures and timeouts; the affected server tests and a database fixture passed in isolation. The subsequent serial `pnpm test:run` was stopped while still running after complete CI passed. Focused local tests, full UI tests, typecheck, and build completed as listed above. - Token gates and `git diff --check` passed. ## Risks - Bundling the Claude bridge increases the server package size. The producer graph retains the qualified SDK version and existing isolation patch. Native platform declarations move to the published server manifest, while the runtime keeps its exact version, declared-slot, no-follow, and executable digest checks. - The new failure category is permanent. Operators must repair the server installation before they retry the task. - Existing failed runs retain their recorded error. The clearer message applies to new runtime tests and runs. - No database schema or migration changes. ## Model Used OpenAI GPT-6 through Codex. The system reports the model family; the exact deployment ID and context-window size are not exposed in this session. Used reasoning, browser inspection, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2223b7de9e |
perf(ci): stop building the standalone packages inside the public-install verifier (#15908)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `Build` check. Its last step is the public npm install verifier, `scripts/verify-grok-npm-install.mjs`, which stages the eighteen public packages reachable from the server and the CLI, packs them, and installs them in a clean consumer. > - Before it stages anything, the verifier runs `scripts/build-standalone-public-packages.mjs`. That script installs and compiles the seven sandbox-provider plugins that live outside the pnpm workspace: one network `pnpm install --ignore-workspace` and one `tsc` per plugin. The line was copied from the build block of `release.sh` when the verifier was written in #14024. > - release.sh needs that build because it publishes those plugins. The verifier does not for anything it stages. Its package closure follows `workspace:` specs from the server and the CLI, and a `workspace:` spec can only name a workspace package, so none of the eighteen packages it packs is a standalone package and no tarball reads that build's output. > - One optional path at the end of the verifier does read it: qualification-package retention, which `docker-runner-check.yml` enables with `PAPERCLIP_RUNNER_QUALIFICATION_PACKAGES_DIR` after running only `pnpm build`, packs the Daytona plugin from its `dist`. The PR `Build` job never sets that variable. > - The timing line from #15868 and #15888 puts the verifier's staging phase at 75.8s on the latest run, of which the staged sources, the bundled installs and the packs account for 46.5s. The rest, about 29s, is the work before staging, and the standalone build is almost all of it. `Build` at 433-530s is the slowest check in most PR runs once the server shards drop, so that time sits on the run's critical path. > - This pull request runs the standalone build only when retention is requested, keeps the `ui-dist` copy the server package needs, and times both on the timing line so what is left before staging is visible. > - The benefit is a `Build` job about 21-28s shorter, with no change to what the verifier installs or checks, no loss of coverage for the standalone build because Canary Dry Run still runs it through `release.sh`, and no change to the retention path that reads it. ## Linked Issues or Issue Description Replaces #15892, which GitHub closed automatically when the branch of #15888 (its base at the time) was deleted on merge. Same two commits, rebased onto `master` on top of #15888 ( |
||
|
|
1ac48aff8d |
perf(ci): pack the verifier's staged packages concurrently (#15888)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `Build` check. Its last step is the public npm install verifier, `scripts/verify-grok-npm-install.mjs`, which stages the eighteen public packages, packs them, and installs them in a clean consumer. > - The verifier packs the eighteen packages one after another with `npm pack`. Each pack is a Node start-up plus gzip of that package's files. The timing line added in #15868 shows this loop at 35.6-37.9s on four `Build` runs, the largest part of the staging phase now that the bundled installs overlap. > - `Build` at 433-530s is the slowest check in most PR runs once the server shards drop, so its staging phase sets the wall-clock time of the run. > - Each pack reads only its own `package-<index>` directory and writes one tarball. The runner has four cores and the packs are CPU-bound. There is no reason to run them one after the other. > - This pull request runs the packs as concurrent children through the prefetch helper, one per core, bundled packages first, each into its own directory, and puts the results back into dependency order before every tarball is checked. > - The benefit is a shorter `Build` job, and a timing line that reports the packs' serial cost next to their wall time, so the saving is measured on this PR's own run instead of estimated. ## Linked Issues or Issue Description Refs #15868, #15861. This branch is stacked on the branch of #15868, which is stacked on #15861, because the same execution branch holds both open PRs. The diff here is this PR's one commit. After #15868 merges, this PR is retargeted to `master` and rebased. **What existing behavior does this improve?** The `Build` PR check. Its public npm install verifier packs the eighteen public packages in one serial loop. **Subsystem affected** CI and release scripts (`scripts/verify-grok-npm-install.mjs`, `scripts/prefetch-child.mjs`). **Current behavior** After the bundled installs are joined, the verifier runs `npm pack --pack-destination <root>` for each of the eighteen packages in turn, finds the new tarball by listing the shared root, and checks it with `tar -tzf`. `stage_pack_ms` was 35.8s, 37.1s, 35.6s and 37.9s on the four runs of #15868. The parent is idle apart from the one pack it waits on, and three of the four cores are idle. **Proposed behavior** Start the packs as children through `startPrefetch`, at most one per core (`os.availableParallelism()`), bundled packages first because they make the largest tarballs. Each child packs into its own `pack-<index>` directory, so the tarball a package produced is read from there instead of guessed from a shared directory. Register each child for cleanup as it starts. Put the results back into dependency order, then check every tarball with `tar -tzf` as before, then copy them into the assets directory as before. Report `stage_pack_check_ms`, `pack_concurrency`, `pack_serial_ms` and `pack_longest_ms` on the timing line, and keep `stage_pack_ms` as the whole step. **Reason and benefit** With four slots the step takes about the longest single pack or a quarter of the serial sum, whichever is larger. The expected saving is 20-25s of the 36-38s, measured on this PR's own `Build` job by `pack_serial_ms` against `stage_pack_ms`. **Measured on this PR's first run ([38146108750](https://github.com/paperclipai/paperclip/actions/runs/38146108750), `Build` 433s, verifier 214s):** `stage_pack_ms` 29.9s against 35.6-37.9s before, so the step shed 6-8s, not the 20-25s estimated. `pack_serial_ms` is 44.4s with `pack_concurrency` 4, and `pack_longest_ms` is 28.3s: one pack (the server with its 513 MB bundled closure, gzip level 9 inside npm) runs for almost the whole step, so the step is bounded by that pack and not by the slot count. The seventeen other packs fit inside it. The serial sum is 7-9s higher than the old serial step, which is the four processes contending for the runner's cores. `stage_packages_ms` 75.8s against 80.4-89.6s on #15868's runs; `image_wait_ms` and `grok_wait_ms` stay 0. **Breaking changes** None. The same packages are packed with the same `npm pack` arguments, the tarballs are checked the same way, and the consumer receives them in the same order. ## What Changed - `scripts/pack-staged-packages.mjs` (new): `packStagedPackages` launches the packs in the order given with at most `concurrency` children at once, through the prefetch helper, and resolves with each entry's tarball path and wall time in the same order. Every child is passed to `register` as it starts. The first failure rejects with the tail of that child's log and nothing further is launched. A child that leaves anything other than one tarball in its directory is a failure. `packConcurrency` is one per core. `packTimings` gives the timing-line keys. - `scripts/verify-grok-npm-install.mjs`: each entry gets its `index` and its own `pack-<index>` destination. The pack loop is replaced by one `packStagedPackages` call, launched bundled-first and registered into the `prefetches` list as each child starts, so an earlier failure stops any pack still running before the scratch root is removed. The results are sorted back into dependency order, every tarball is checked in that order, and the copy into the assets directory is unchanged. Only the lifecycle sentinel is still packed in-process. - Timing line: `stage_pack_check_ms`, `pack_concurrency`, `pack_serial_ms` and `pack_longest_ms` are added. `stage_pack_ms` still covers the whole step, so the before and after numbers compare directly. - `scripts/__tests__/pack-staged-packages.test.mjs` (new): five tests with a local stand-in for npm and no network. The default concurrency and the pack arguments; four packs with two slots run the two slow packs together, start a third only once a slot is free, receive the exact `npm pack` argument list, register every child, and come back in entry order with their tarballs and timing keys although the fast packs finish first; a failed pack rejects with its log tail, launches nothing more, and leaves the running pack to be stopped; a pack that produces two tarballs or none is a failure; and no entries resolve at once. - `scripts/__tests__/grok-public-install-sandbox.test.mjs`: the guard test on the verifier source now checks the import, the bundled-first launch with cleanup registration, the reorder before the checks, the checks before the copy, that only the sentinel is packed in-process, that each pack has its own directory, and the new timing keys. - `package.json`: the new test file is in `test:release-registry`. ## Verification - `node --test scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/prefetch-child.test.mjs`: 35 tests pass (30 on #15868's head plus the five new ones). The new file passes three consecutive runs. - `node --check` passes on both scripts. - With the real npm 11.12.1 on Node 25.9: two small packages packed through `packStagedPackages` with two slots produce one tarball each in their own `pack-<index>` directory, with the expected `package/` listing from `tar -tzf`. - The verifier itself needs Docker and a network, which this workspace does not have. This PR's own `Build` job runs the branch's verifier; its timing line carries `stage_pack_ms`, `pack_serial_ms`, `pack_longest_ms` and `pack_concurrency`, which is the measurement. Before, on #15868's four runs: `stage_pack_ms` 35.8s / 37.1s / 35.6s / 37.9s. - `docker-runner-check.yml` runs the same script after `pnpm build` and needs no change. ## Risks - Four `npm pack` processes run at once and share the host npm configuration and cache. `npm pack --ignore-scripts` reads the package directory and writes one tarball; it does not write to the cache beyond its per-process log file. If a shared-state race appears, the failing pack's log tail is in the error and the fix is to lower the concurrency. - Memory and CPU: four Node processes instead of one on the four-core `Build` runner, for about thirty seconds. The largest pack (the server with its bundled dependencies) is the same process as before. - The pack output moves from the CI log into per-pack log files in the scratch root. A failure reports the last 2000 characters of the failing pack's log; a successful pack's output is discarded with the scratch root. - The tarball checks (`tar -tzf`) now run after all packs instead of after each pack. A bundled Codex payload is still rejected before any tarball reaches the consumer; the only change is that the error arrives a few seconds later. - Low risk otherwise. The bundled installs, the consumer install, and every probe are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.1 (Anthropic), model ID `claude-fable-5-1`, run as a Paperclip agent through the Claude Code local adapter with tool use (shell, file edits, GitHub CLI). Reasoning mode: extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (run 38146108750: `server (11/12)` failed on its first attempt on a teardown `EACCES` while removing the per-worker `PAPERCLIP_HOME` after all 74 files and 1257 tests had passed, in a file this PR does not touch; the re-run of that job passed and every check is green) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
026bdcd046 |
perf(ci): run the verifier's bundled-dependency installs concurrently (#15868)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every pull request runs the `Build` check. Its last step is the public npm install verifier, `scripts/verify-grok-npm-install.mjs`, which stages the eighteen public packages and installs them in a clean consumer. > - The verifier stages the packages in one serial loop. Three packages bundle dependencies (`@paperclipai/server`, `@paperclipai/adapter-utils`, `@paperclipai/db`). Each one runs a host-side `npm install` of its bundled closure in-process, one after the other, and nothing in the loop overlaps anything else. The staging phase took 102-116s on recent `Build` jobs. > - `Build` at 433-530s is the slowest check in most PR runs once the server shards drop, so its staging phase sets the wall-clock time of the run. > - The three installs write only their own directories and read nothing from each other. There is no reason to run them one after the other. > - This pull request starts the three installs together as child processes and copies the other fifteen packages while they run. Packing stays serial and in order. > - The benefit is a shorter `Build` job, and a timing line that reports each install, so the saving is measured on this PR's own run instead of estimated. ## Linked Issues or Issue Description Refs #15861. This branch is stacked on the branch of #15861 because the same execution branch holds that open PR. The diff here is this PR's three commits. After #15861 merges, this PR is retargeted to `master` and rebased. **What existing behavior does this improve?** The `Build` PR check. Its public npm install verifier stages the eighteen public packages in one serial loop. **Subsystem affected** CI and release scripts (`scripts/verify-grok-npm-install.mjs`, `scripts/prepare-bundled-package.mjs`, `scripts/prefetch-child.mjs`, `scripts/prefetch-supervisor.mjs`). **Current behavior** The staging loop calls `prepareBundledPackage` in-process for the three packages that bundle dependencies. Each call runs a host-side `npm install` of that package's bundled closure, applies the repo patches, and rewrites the manifest, before the loop moves to the next package. The timing line reports the whole staging phase as one number, 102-116s on recent runs. The three installs are the only network work in that phase. **Proposed behavior** Stage the three bundled packages' sources first, start their installs at the same time as children of the script's own CLI entry (the same path `release.sh` uses), copy the other fifteen packages while the installs run, join the installs, and then pack every package in dependency order as before. Report `stage_sources_ms`, `stage_bundled_ms`, `stage_pack_ms`, `bundled_wait_ms` and one `stage_bundled_<package>_ms` per install on the timing line. **Reason and benefit** The staging phase shortens by the two shorter installs. Measured on this PR's first run (below): the three installs sum to 25.9s and take 16.4s of wall time together, so the overlap removes about 9.5s from the phase, and the timing line now shows that packing (35.8s) and the standalone-package build before the loop (about 27s) are the larger remaining costs. The original estimate of 30-50s was too high because it assumed the installs were about half of the phase; they are a quarter. **Breaking changes** None. The packed tarballs are produced by the same script, in the same order, and are checked the same way. ## What Changed - `scripts/stage-bundled-packages.mjs` (new): `startBundledInstalls` starts one child per bundled package through the prefetch helper from #15849, with the child's output in a log file. `bundledInstallTimings` reads each child's wall time for the timing line. `bundledPackageSlug` maps a package name to its log and timing key. - `scripts/verify-grok-npm-install.mjs`: the staging loop is split. The bundled packages' sources are staged first and their installs start together. The other packages are copied while the installs run. The install handles are pushed into the `prefetches` list at once, so an earlier failure stops any install still running before the scratch root is removed. The join comes before the first `npm pack`. The pack loop walks every package in `needed` order and checks each tarball as before. `prepareBundledPackage` is no longer imported. - Timing line: `stage_sources_ms`, `stage_bundled_ms`, `stage_pack_ms`, `bundled_wait_ms` and `stage_bundled_server_ms`, `stage_bundled_adapter_utils_ms`, `stage_bundled_db_ms` are added. `stage_packages_ms` and the download keys are unchanged, so the before and after numbers compare directly. - `scripts/__tests__/stage-bundled-packages.test.mjs` (new): five tests with local Node children and no network. The slug mapping; the children run the CLI entry `release.sh` runs and never an in-process call; two installs run at the same time, each with its own source and target, and their logs and timing keys are correct (each child exits only after the other has started, so serial children would fail); a failed install rejects with its own log tail while the other is still running and is then stopped; and stopping the installs after one fails also stops the stand-in npm process the other wrapper is blocked on. - `scripts/__tests__/grok-public-install-sandbox.test.mjs`: one guard test on the verifier source. The bundled sources are staged before the installs start, the installs are registered for cleanup before the other packages are copied, the join comes before the pack loop, the pack loop is in order, there is still exactly one package pack loop, and the timing keys are reported. - `package.json`: the new test file is in `test:release-registry`. ### Review fix (second commit): stop a prefetch's whole process tree Greptile pointed out that `stopPrefetches` signalled only the Node wrapper of each install, while the wrapper is blocked in `execFileSync('npm', ['install', ...])`. After one install failed, the other wrapper died and its npm process kept writing into the scratch root the verifier was about to remove. - `scripts/prefetch-child.mjs`: on POSIX every prefetch child is started `detached`, so it leads its own process group. `stopPrefetches` signals the group (`process.kill(-pid, ...)`) and waits until no member of the group exists, with `SIGKILL` after the grace period and a second grace period before it gives up so cleanup cannot hang. On Windows the child is signalled directly as before. - `scripts/__tests__/prefetch-child.test.mjs`: two tests. A stopped prefetch's grandchild is gone after `stopPrefetches` returns, and a `SIGINT` to a separate parent process reaches the grandchild while the parent still exits with `SIGINT`. Both are skipped on Windows. - `scripts/__tests__/stage-bundled-packages.test.mjs`: one test, the cancellation case Greptile asked for. The stand-in `prepare-bundled-package` blocks in `execFileSync` on a stand-in npm that announces its pid; after the other install fails, `stopPrefetches` must leave that pid gone. All three new tests fail against the previous helper (`grandchild <pid> outlived stopPrefetches`, `outlived the parent's SIGINT`, `stand-in npm <pid> outlived stopPrefetches`) and pass with the fix. ### Review fix (third commit): the parent runs no signal handler Greptile pointed out that the second commit forwarded the terminal's signals to the detached groups through a JavaScript handler in the parent. The verifier spends most of its time blocked in `execFileSync` (the standalone builds, every `npm pack`), where no handler can run, and installing the handler also replaced the parent's immediate default termination. So a Ctrl-C during a synchronous phase did nothing until that command returned, and the detached installs and downloads kept running meanwhile. Confirmed with the new test against the second commit's helper: the parent survived the `SIGINT`, exited with code 1 once the synchronous command had died, and its handler never ran. - `scripts/prefetch-supervisor.mjs` (new): every prefetch command now runs under this small Node script as the leader of its own process group. The supervisor holds a pipe from the parent on stdin (which the parent writes to only to request a stop where it cannot signal the group; see the fourth commit). The kernel closes it when the parent exits for any reason, including a signal delivered while the parent is blocked, or `SIGKILL`, and the supervisor then stops its group (`SIGTERM`, `SIGKILL` after the grace period) and says so in the log. It passes group-wide signals on to the command and stays alive until the command exits, so its own exit reports the command's exit code or signal. Run by hand with a terminal or `/dev/null` on stdin it watches nothing. - `scripts/prefetch-child.mjs`: `startPrefetch` spawns the supervisor with the command, with stdin as the pipe. The forwarding handlers are gone; the parent installs no signal handler, so its default termination stays immediate. `stopPrefetches` is unchanged. - `scripts/__tests__/prefetch-child.test.mjs`: the first test asserts the parent has no `SIGINT`/`SIGTERM`/`SIGHUP` listener after a prefetch starts. The "signal to the parent" test now runs for `SIGINT` and `SIGKILL` and checks the supervisor's log line. One new test gives a separate parent its own process group, blocks it in `execFileSync` on a 20s command, and sends `SIGINT` to that group the way a terminal sends Ctrl-C: the parent must die of `SIGINT` within two seconds and the prefetch's grandchild must be gone. Against the previous helper the parent exits with code 1 instead, the grandchild outlives `SIGKILL`, and the handler-count assertion fails. ### Review fix (fourth commit): a deliberate stop on Windows goes through the pipe Greptile pointed out that on Windows, where `startPrefetch` now returns the supervisor, `stopPrefetches` called `child.kill('SIGTERM')` on it. Windows has no process groups and `kill` there ends a process outright, so the supervisor died without running its handler and its command kept running after cleanup returned. - `scripts/prefetch-child.mjs`: without process groups, the graceful stop no longer signals the supervisor. `stopPrefetches` writes a stop request to the pipe the supervisor already watches for the parent's exit (`stdin.end('stop\n')`). Only the `SIGKILL` escalation targets the supervisor itself, after it has had its grace period. POSIX keeps the group signal, so `stopPrefetches` and the existing stop and failure tests are unchanged there. - `scripts/prefetch-supervisor.mjs`: data on stdin is a stop request and EOF is the parent's exit; both end the group the same way (`SIGTERM`, `SIGKILL` after the grace period) and the log line names which one it was. - `scripts/__tests__/prefetch-child.test.mjs`: one new test writes the stop request to a running prefetch's pipe instead of calling `stopPrefetches`. The supervisor must exit with `SIGTERM`, the command's grandchild must be gone, and the log must say `stop requested` and not `parent exited`. The previous supervisor fails it on the log line, since it treated the request as the parent's exit. The Windows kill itself cannot be reproduced on the Linux runners, so the Windows branch of `signalTree` is verified by that supervisor test plus reading its two lines. The stand-in that ignores `SIGTERM` is now skipped on Windows, where `kill` cannot be ignored. ## Verification - `node --test scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs`: 30 tests pass (was 19 before this PR, 24 after the first commit, 27 after the second, 29 after the third). The three files pass three consecutive runs. - `node --check` passes on all four scripts. - The verifier itself needs Docker and a network, which this workspace does not have. This PR's own `Build` job runs the branch's verifier. Measured on all four commits' runs ([38132868803](https://github.com/paperclipai/paperclip/actions/runs/38132868803): `Build` 428s, `pnpm build` 112s, verifier step 214s; [38136912842](https://github.com/paperclipai/paperclip/actions/runs/38136912842): `Build` 447s; [38139871529](https://github.com/paperclipai/paperclip/actions/runs/38139871529): `Build` 432s; [38142762538](https://github.com/paperclipai/paperclip/actions/runs/38142762538): `Build` 457s, `pnpm build` 123s, verifier step 231s): | Key | Run 1 | Run 2 | Run 3 | Run 4 | Before | | --- | --- | --- | --- | --- | --- | | `stage_packages_ms` | 80.4s | 84.9s | 81.5s | 89.6s | 102-116s on master runs 38112764439, 38111191543, 38110932348, 38110914319, 38110897745, 38127611057; 84.5s on #15849's run on a runner as fast as these | | `stage_sources_ms` | 1.4s | 0.7s | 0.9s | 0.9s | (inside the number above) | | `stage_bundled_server_ms` / `adapter_utils` / `db` | 16.3s / 5.0s / 4.6s | 14.9s / 3.6s / 3.3s | 16.4s / 5.2s / 4.7s | 17.1s / 4.4s / 3.8s | ran one after another, 21.8-25.9s together | | `stage_bundled_ms` (wall, all three) | 16.4s | 15.0s | 16.5s | 17.2s | 21.8-25.9s | | `bundled_wait_ms` | 15.3s | 14.7s | 16.1s | 16.7s | — | | `stage_pack_ms` | 35.8s | 37.1s | 35.6s | 37.9s | (inside the number above) | | `consumer_install_ms` | 94.4s | 97.8s | 95.7s | 99.8s | 97-134s | | `image_prefetch_ms` / `image_wait_ms` | 33.7s / 0ms | 31.8s / 0ms | 34.7s / 0ms | 33.7s / 0ms | unchanged | | `grok_prefetch_ms` / `grok_wait_ms` | 2.1s / 0ms | 2.1s / 0ms | 2.0s / 0ms | 2.1s / 0ms | unchanged | The overlap removed about 7-10s from the staging phase on each of the four runs (21.8-25.9s of installs in 15.0-17.2s of wall time). The third and fourth commits changed no timing: both waits stay at 0ms and the staging phase sits inside the band of the first two runs. The rest of the phase is the standalone-package build and the server UI dist prep before the loop (about 27-32s) and the eighteen serial `npm pack` plus `tar -tzf` checks (36-37s). - `docker-runner-check.yml` runs the same script after `pnpm build` and needs no change. ## Risks - Three `npm install` processes share the host npm cache at the same time. npm's cache (`cacache`) is designed for concurrent writers, and the three installs have disjoint target directories. If a shared-cache race appears, the failing install's log tail is in the error and the fix is to serialize again. The first CI run showed none. - The install output moves from the CI log into per-install log files in the scratch root. A failure reports the last 2000 characters of the failing install's log; a successful install's output is discarded with the scratch root. - Memory: three concurrent installs on the `Build` runner instead of one. The closures are small (the server's eight bundled packages are the largest), so this stays well under the runner's memory. - Process groups: a detached child no longer receives the terminal's signals on its own. The supervisor stops its group when the parent is gone, which it learns from the kernel closing the stdin pipe, so this holds for a Ctrl-C while the parent is blocked in synchronous work and for `SIGKILL` alike. The parent keeps Node's default signal handling. If the supervisor itself is killed with `SIGKILL` before its command, the command is orphaned and finishes on its own (each install ends within seconds; the image pull within a minute), the same as any orphaned grandchild before this PR. - One extra Node process per prefetch (five per verifier run). Each costs about 40ms to start, inside the time the parent spends staging anyway. - Low risk otherwise. Packing, the tarball checks, the consumer install, and every probe are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.1 (Anthropic), model ID `claude-fable-5-1`, run as a Paperclip agent through the Claude Code local adapter with tool use (shell, file edits, GitHub CLI). Reasoning mode: extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on all four commits - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (on the fourth commit Greptile reviewed 8 files and added no comments; all three earlier threads are resolved) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Bender (Fable) <bender-fable@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> |
||
|
|
42ecc96556 |
perf(ci): prefetch the consumer image and Grok binary during package staging (#15849)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pull requests to this repository run a large CI matrix. The slowest check sets the time a contributor waits for a green run. > - After #15826 leveled the server shards and #15831 split `workspaces-b`, the `Build` lane is the slowest check: 634s in run 38107578322 (2026-10-11 03:11Z), ahead of the next server shard at 574s. It was 533s in run 38107430861 and 543s to 568s in earlier runs that night. > - `Build` is about 100s of setup, 148s to 195s of `pnpm build`, and 278s to 320s of `scripts/verify-grok-npm-install.mjs`. The verifier stages and packs the public packages first (84s to 100s of CPU work), and only then pulls the pinned `node:24-trixie` consumer image (8 layers, 440 MB compressed) and downloads the Grok binary (65 MB gzip, 166 MB verified). Both network waits sit after the staging work on the critical path. > - Neither download depends on the packed packages. The image is pinned by digest and the Grok binary is checked against its pinned sha256 by the provisioning script, so starting them early changes no verified property. > - This pull request starts both downloads as background children before staging and joins them where they are needed: the image before the first sandbox, the binary after the negative probe. It also prints one timing line per phase so the next slow part is visible in the CI log. > - The benefit is that the whole pull and the whole Grok download hide behind the staging phase. The verifier should shed the full pull time on a fresh runner, and the next cycle can read the phase split instead of inferring it from log gaps. ## Linked Issues or Issue Description **Describe the enhancement** The `Build` PR check is now the slowest check in the run. Its last step, the public npm install verifier, pulls the consumer Docker image and downloads the Grok binary only after it has finished staging and packing the eighteen public packages. Both downloads could run while the packages are staged. **Why is this needed** The slowest check decides the wall-clock time of every PR run. In the most recent fully green run (38107578322), `Build` took 634s: `pnpm build` 195s and the verifier 320s. The verifier's log shows the staging phase ends at +100s and the Grok result arrives at +267s. The image pull and the Grok download are inside that second window, in series with the consumer install and the probes. **Proposed solution** Start `docker pull --platform linux/amd64 <pinned image>` and `provision-grok.mjs` as background children at the top of the verifier, before the staging loop. Await the image right before the first sandbox call. Provision Grok into a prefetch directory and rename it into `native/grok` only after the "npm must not provision Grok" assertion and the negative probe, so that assertion keeps its meaning and only the positive probe sees the binary. Print a `paperclip.public-npm-install.timing.v1` line with wall time per phase and the time the main flow waited on each prefetch. **Alternatives considered** - Run the verifier as its own job in parallel with `Build`. It needs the full `pnpm build` output, so that job would be as long as `Build` is now. - Warm the consumer npm cache from an Actions cache. That changes the "clean public npm install" property the verifier reports. Not done here; the new timing line will show whether the consumer install is worth that discussion. - Pack the eighteen packages concurrently. The three bundled packages run their own npm installs in series inside `prepareBundledPackage`, which is synchronous. A larger restructure; left for a later cycle with timing data. - Overlap the Pi runtime setup with the Grok probes. Both use the same consumer directory from separate containers, and the probes after provisioning are short. Not worth the concurrency on shared state. ## What Changed - `scripts/prefetch-child.mjs` (new): `startPrefetch({ name, cmd, args, cwd, env, logPath })` spawns a background child with no inherited terminal and its stdout and stderr in a log file, and returns `{ name, child, done, durationMs }`. `done` resolves on exit 0 and otherwise rejects with the tail of the log; the rejection is observed inside the helper, so an unawaited failure never becomes an unhandled rejection. `durationMs` ends at the child's last write to its log (the exit event is observed late when the caller is blocked in synchronous work; both downloads print their final status line as they complete). `stopPrefetches(prefetches, { graceMs })` sends `SIGTERM` to every child still running, escalates to `SIGKILL` after the grace period, and resolves only once all of them have exited. - `scripts/verify-grok-npm-install.mjs`: the image pull and the Grok provisioning start through the helper inside the `try` block, before the staging loop. The main flow awaits the image before the first `npm install` sandbox and awaits Grok after the negative probe, then renames the staged binary into `native/grok` before the positive probe. The `finally` block awaits `stopPrefetches` and only then removes the scratch root, so cleanup never races a child's writes. The script prints one timing JSON line after the Pi probe, with the per-phase wall times, the two `*_wait_ms` values, and the corrected `*_prefetch_ms` values. - `scripts/__tests__/prefetch-child.test.mjs` (new): seven tests run small local Node children, with no Docker or network: a successful prefetch resolves and keeps its output; the duration ends at the download even when the parent is blocked for 1.5s; a nonzero exit rejects with the log tail; a command that cannot start rejects instead of hanging the join or the stop; stopping a running child waits for its exit; a child that ignores `SIGTERM` is killed after the grace period; an unawaited failure does not produce an unhandled rejection in a fresh process. - `scripts/__tests__/grok-public-install-sandbox.test.mjs`: the source test asserts that exactly the pinned image is pulled for `linux/amd64`, that Grok is provisioned into the prefetch directory and never straight into `native/grok`, that both prefetches start before staging, that the image is joined before the first sandbox, that the rename happens after the npm assertion and the negative probe and before the positive probe, that the verifier imports the helper, and that `stopPrefetches` is awaited before the single `rmSync(root, ...)`. - `package.json`: `test:release-registry` runs the new test file. ## Verification - `node --test scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs`: 22 pass (7 new in `prefetch-child.test.mjs`, 12 in the sandbox file). The new file passed on three consecutive runs. - `node --check scripts/verify-grok-npm-install.mjs` passes. - `node packages/paperclip-runner/scripts/provision-grok.mjs <scratch>/prefetch/grok` on this host: `Verified Grok 1.0.13 (linux-x64)` in 1.0s, `native/grok` absent before the move, 166079904 bytes in place after it. - Docker is not installed on the host that produced this PR, so the verifier was not run end to end locally. This PR's own `Build` job runs `node scripts/verify-grok-npm-install.mjs` from the branch checkout, so it exercises the change in full. **Measured on this PR's first CI run** ([38108964834](https://github.com/paperclipai/paperclip/actions/runs/38108964834), created 03:37Z), against two `master` runs created in the same twelve minutes: | | 38108878166 (`master`, 03:35Z) | 38108995800 (`master`, 03:37Z) | 38108964834 (this PR, 03:37Z) | | --- | --- | --- | --- | | Verifier step | 307s | 248s | 219s | | `Build` job | 555s | 484s | 459s | | Slowest check in the run | `Build` 555s | server (8/12) 535s | server (12/12) 547s | The PR run's timing line: `stage_packages_ms 84464`, `image_wait_ms 1`, `consumer_install_ms 97305`, `lifecycle_rebuild_ms 3545`, `codex_probe_ms 1211`, `grok_wait_ms 0`, `grok_probes_ms 626`, `pi_setup_ms 22194`, `pi_probe_ms 7753`. Both waits are at most 1ms, so the staging phase hid both downloads in full. The window from the end of staging to the Grok result went from 161s to 167s on the two runs that motivated this PR to 102s here, so the hidden downloads were about 60s on a fresh `runs-on` spot instance. Against the two same-window `master` runs, the verifier step is 29s to 88s shorter and `Build` is 25s to 96s shorter; the spread is runner variance in `pnpm build` and the consumer install. The `image_prefetch_ms` and `grok_prefetch_ms` values on that first run (84550 and 186696) were the parent's late observation of each exit, not download times; that is the first Greptile finding, fixed by the second commit, which measures from the download's last output instead. ## Risks - Low risk. The verifier proves the same properties in the same order. The only ordering change is that the downloads start earlier; every assertion, probe, and sandbox argument is unchanged. - The Grok binary is written under the scratch root in a `prefetch` directory during staging. That directory is never mounted into a container, and the `native/grok` assertion still runs against an absent path. - A prefetch that fails early now fails the step at its join point, where the old code would have failed at the same download a few seconds later. Same outcome, same message source. - `*_prefetch_ms` is a last-output time. A child that printed early and then kept working silently would be under-reported; neither `docker pull` nor `provision-grok.mjs` behaves that way (both print their final status as they finish), and a silent child falls back to the observed exit time. - Cleanup now waits up to 5s for a child that ignores `SIGTERM` before killing it, so a failing run can end up to 5s later than before. Both children exit promptly on `SIGTERM` in practice. - The timing line is one more stdout line. Nothing in the repository parses the verifier's stdout; the two workflows that run it write the output to the job log or to a file. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, running as a Paperclip agent through the Claude local adapter with tool use (shell, file edits, GitHub CLI). The agent collected the CI data, made the change, ran the tests, and wrote this description. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
8dd381f5dd |
perf(ci): run the db project as its own workspaces-c lane (#15831)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Pull requests to this repository run a large CI matrix. The slowest
check sets the time a contributor waits for a green run.
> - The `General tests (workspaces-b)` lane runs every non-server Vitest
project except `ui` and `cli` in one job, one project after the other.
After #15826 leveled the server shards, this lane became the slowest
check: 593s in run 38084171761, 523s in 38083084276 and 499s in
38083020834 (all 2026-10-10).
> - The `@paperclipai/db` project alone is 168s to 250s of that lane's
436s to 513s of vitest time. Its `client.test.ts` suite is 117s to 189s
and `backup-lib.test.ts` is 70s to 107s. The other seventeen projects
together are only 263s to 276s.
> - Vitest's native `--shard` does not level this lane. The db project
is already file-parallel, and a hash-ordered slice can put both slow
suites on one runner.
> - This pull request moves the db project into its own `workspaces-c`
lane in both `pr-trusted.yml` and `release-verify.yml`. The runner
script gains a `general-workspaces-c` group, and `general-workspaces-b`
drops the db project.
> - The benefit is that both lanes should finish in about 230s to 340s
instead of 499s to 593s, and `workspaces-b` leaves the critical path.
The cost is one more job per run, about 60s of setup.
## Linked Issues or Issue Description
**Describe the enhancement**
The `General tests (workspaces-b)` PR check is the slowest check in the
run. It runs seventeen Vitest projects in sequence on one runner. The
`@paperclipai/db` project is one third to one half of its vitest time.
**Why is this needed**
The slowest check decides the wall-clock time of every PR run. In the
last three fully green runs on the current server-shard manifest,
`workspaces-b` took 499s, 523s and 593s (runs 38083020834, 38083084276
and 38084171761, 2026-10-10). The next slowest lanes, `Build` and the
server shards, sit at about 480s to 570s. A split of `workspaces-b`
removes 170s to 250s from this lane.
**Proposed solution**
Run the `@paperclipai/db` project as its own lane,
`general-workspaces-c`. Keep the other seventeen projects in
`general-workspaces-b`. Add the lane to both the PR and the release
verification matrix. Extend the guard tests so the three workspace lanes
still cover the root Vitest project list exactly once, so the new group
rejects shard flags, and so both workflows carry the lane unsharded.
**Alternatives considered**
- Vitest native `--shard` on `workspaces-b`, as `workspaces-a` does. The
db project is already file-parallel and its wall time is set by two long
suites, so a hash-ordered slice does not give a predictable split.
- Move `adapter-utils` (115s to 117s) into the new lane as well. That
makes the db lane 285s to 366s of tests and the rest 147s to 160s, which
is less level than db alone (168s to 250s) against the rest (263s to
276s).
- Three lanes. Not needed now: after this split both lanes sit under the
server shards and `Build`, so a third runner would not shorten the run.
## What Changed
- `scripts/run-vitest-stable.mjs`: new `general-workspaces-c` group that
runs `@paperclipai/db` alone. `general-workspaces-b` now excludes it.
Local `--mode all` runs all four general groups, so every project still
runs once. The dry-run output reports the new group's projects.
- `.github/workflows/pr-trusted.yml`: a `workspaces-c` entry in the
general tests matrix, with a comment that records the 2026-10-10
measurements.
- `.github/workflows/release-verify.yml`: the same `workspaces-c` entry,
to keep parity with the PR workflow.
- `scripts/__tests__/run-vitest-stable-shard.test.mjs`: the coverage
test includes the new group; shard flags are rejected for both
`workspaces-b` and `workspaces-c`; a new test asserts that
`workspaces-c` runs only the db project, that `workspaces-b` no longer
carries it, and that both workflows list both lanes without shard
arguments. The workflow check parses each `- group:` matrix entry with
comment lines removed, so a comment between keys cannot hide a shard
setting, and it first confirms the parser sees the shard keys on the
`workspaces-a` entries. A small parser test covers the
comment-between-keys case (Greptile P2).
- `scripts/__tests__/release-verify-workflow.test.mjs`: the release
workflow must carry `general-workspaces-c`.
## Verification
- `GITHUB_WORKFLOW= node --test
scripts/__tests__/run-vitest-stable-shard.test.mjs`: 32 pass (was 29).
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`: the
workflow parity test passes. Two unrelated tests in that file (`a
publisher lockfile lets a fresh smoke install retain validation after
... drift`) fail on my machine on pristine `master` too, because `pnpm`
is not on its PATH. They pass in CI.
- `node scripts/run-vitest-stable.mjs --mode general --group
general-workspaces-c --dry-run` reports `["@paperclipai/db"]`; `--group
general-workspaces-b` reports the other seventeen projects.
- Both workflow files parse as YAML; the general tests matrix has 19
entries in `pr-trusted.yml` and 17 in `release-verify.yml`.
- This PR's own run exercises the script change but not the workflow
change: `pr.yml` calls `pr-trusted.yml@master`, so the matrix comes from
`master` and the `workspaces-c` job appears only on PR runs created
after the merge. Because the script comes from this branch, the
`workspaces-b` job in this PR's run (38090713637) ran the seventeen
remaining projects and finished in 340s, inside the expected 325s to
340s band. No job in that run ran `@paperclipai/db`.
- To cover the gap, I ran the new lane locally with the same command CI
uses, `pnpm test:run:general -- --group general-workspaces-c`: 49 test
files, 180 tests, all passed, 111s of vitest time (
|