Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-12 08:19:54 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
1ac48aff8dc13926539b085ca2e1d67913bcf7dc
PaperClipAI/package.json
T
Devin FoleydevinfoleyClaude Fable 5.1
 1ac48aff8d perf(ci): pack the verifier's staged packages concurrently (#15888) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the `Build` check. Its last step is the
public npm install verifier, `scripts/verify-grok-npm-install.mjs`,
which stages the eighteen public packages, packs them, and installs them
in a clean consumer.
> - The verifier packs the eighteen packages one after another with `npm
pack`. Each pack is a Node start-up plus gzip of that package's files.
The timing line added in #15868 shows this loop at 35.6-37.9s on four
`Build` runs, the largest part of the staging phase now that the bundled
installs overlap.
> - `Build` at 433-530s is the slowest check in most PR runs once the
server shards drop, so its staging phase sets the wall-clock time of the
run.
> - Each pack reads only its own `package-<index>` directory and writes
one tarball. The runner has four cores and the packs are CPU-bound.
There is no reason to run them one after the other.
> - This pull request runs the packs as concurrent children through the
prefetch helper, one per core, bundled packages first, each into its own
directory, and puts the results back into dependency order before every
tarball is checked.
> - The benefit is a shorter `Build` job, and a timing line that reports
the packs' serial cost next to their wall time, so the saving is
measured on this PR's own run instead of estimated.

## Linked Issues or Issue Description

Refs #15868, #15861. This branch is stacked on the branch of #15868,
which is stacked on #15861, because the same execution branch holds both
open PRs. The diff here is this PR's one commit. After #15868 merges,
this PR is retargeted to `master` and rebased.

**What existing behavior does this improve?**

The `Build` PR check. Its public npm install verifier packs the eighteen
public packages in one serial loop.

**Subsystem affected**

CI and release scripts (`scripts/verify-grok-npm-install.mjs`,
`scripts/prefetch-child.mjs`).

**Current behavior**

After the bundled installs are joined, the verifier runs `npm pack
--pack-destination <root>` for each of the eighteen packages in turn,
finds the new tarball by listing the shared root, and checks it with
`tar -tzf`. `stage_pack_ms` was 35.8s, 37.1s, 35.6s and 37.9s on the
four runs of #15868. The parent is idle apart from the one pack it waits
on, and three of the four cores are idle.

**Proposed behavior**

Start the packs as children through `startPrefetch`, at most one per
core (`os.availableParallelism()`), bundled packages first because they
make the largest tarballs. Each child packs into its own `pack-<index>`
directory, so the tarball a package produced is read from there instead
of guessed from a shared directory. Register each child for cleanup as
it starts. Put the results back into dependency order, then check every
tarball with `tar -tzf` as before, then copy them into the assets
directory as before. Report `stage_pack_check_ms`, `pack_concurrency`,
`pack_serial_ms` and `pack_longest_ms` on the timing line, and keep
`stage_pack_ms` as the whole step.

**Reason and benefit**

With four slots the step takes about the longest single pack or a
quarter of the serial sum, whichever is larger. The expected saving is
20-25s of the 36-38s, measured on this PR's own `Build` job by
`pack_serial_ms` against `stage_pack_ms`.

**Measured on this PR's first run
([38146108750](https://github.com/paperclipai/paperclip/actions/runs/38146108750),
`Build` 433s, verifier 214s):** `stage_pack_ms` 29.9s against 35.6-37.9s
before, so the step shed 6-8s, not the 20-25s estimated.
`pack_serial_ms` is 44.4s with `pack_concurrency` 4, and
`pack_longest_ms` is 28.3s: one pack (the server with its 513 MB bundled
closure, gzip level 9 inside npm) runs for almost the whole step, so the
step is bounded by that pack and not by the slot count. The seventeen
other packs fit inside it. The serial sum is 7-9s higher than the old
serial step, which is the four processes contending for the runner's
cores. `stage_packages_ms` 75.8s against 80.4-89.6s on #15868's runs;
`image_wait_ms` and `grok_wait_ms` stay 0.

**Breaking changes**

None. The same packages are packed with the same `npm pack` arguments,
the tarballs are checked the same way, and the consumer receives them in
the same order.

## What Changed

- `scripts/pack-staged-packages.mjs` (new): `packStagedPackages`
launches the packs in the order given with at most `concurrency`
children at once, through the prefetch helper, and resolves with each
entry's tarball path and wall time in the same order. Every child is
passed to `register` as it starts. The first failure rejects with the
tail of that child's log and nothing further is launched. A child that
leaves anything other than one tarball in its directory is a failure.
`packConcurrency` is one per core. `packTimings` gives the timing-line
keys.
- `scripts/verify-grok-npm-install.mjs`: each entry gets its `index` and
its own `pack-<index>` destination. The pack loop is replaced by one
`packStagedPackages` call, launched bundled-first and registered into
the `prefetches` list as each child starts, so an earlier failure stops
any pack still running before the scratch root is removed. The results
are sorted back into dependency order, every tarball is checked in that
order, and the copy into the assets directory is unchanged. Only the
lifecycle sentinel is still packed in-process.
- Timing line: `stage_pack_check_ms`, `pack_concurrency`,
`pack_serial_ms` and `pack_longest_ms` are added. `stage_pack_ms` still
covers the whole step, so the before and after numbers compare directly.
- `scripts/__tests__/pack-staged-packages.test.mjs` (new): five tests
with a local stand-in for npm and no network. The default concurrency
and the pack arguments; four packs with two slots run the two slow packs
together, start a third only once a slot is free, receive the exact `npm
pack` argument list, register every child, and come back in entry order
with their tarballs and timing keys although the fast packs finish
first; a failed pack rejects with its log tail, launches nothing more,
and leaves the running pack to be stopped; a pack that produces two
tarballs or none is a failure; and no entries resolve at once.
- `scripts/__tests__/grok-public-install-sandbox.test.mjs`: the guard
test on the verifier source now checks the import, the bundled-first
launch with cleanup registration, the reorder before the checks, the
checks before the copy, that only the sentinel is packed in-process,
that each pack has its own directory, and the new timing keys.
- `package.json`: the new test file is in `test:release-registry`.

## Verification

- `node --test scripts/__tests__/pack-staged-packages.test.mjs
scripts/__tests__/grok-public-install-sandbox.test.mjs
scripts/__tests__/stage-bundled-packages.test.mjs
scripts/__tests__/prefetch-child.test.mjs`: 35 tests pass (30 on
#15868's head plus the five new ones). The new file passes three
consecutive runs.
- `node --check` passes on both scripts.
- With the real npm 11.12.1 on Node 25.9: two small packages packed
through `packStagedPackages` with two slots produce one tarball each in
their own `pack-<index>` directory, with the expected `package/` listing
from `tar -tzf`.
- The verifier itself needs Docker and a network, which this workspace
does not have. This PR's own `Build` job runs the branch's verifier; its
timing line carries `stage_pack_ms`, `pack_serial_ms`, `pack_longest_ms`
and `pack_concurrency`, which is the measurement. Before, on #15868's
four runs: `stage_pack_ms` 35.8s / 37.1s / 35.6s / 37.9s.
- `docker-runner-check.yml` runs the same script after `pnpm build` and
needs no change.

## Risks

- Four `npm pack` processes run at once and share the host npm
configuration and cache. `npm pack --ignore-scripts` reads the package
directory and writes one tarball; it does not write to the cache beyond
its per-process log file. If a shared-state race appears, the failing
pack's log tail is in the error and the fix is to lower the concurrency.
- Memory and CPU: four Node processes instead of one on the four-core
`Build` runner, for about thirty seconds. The largest pack (the server
with its bundled dependencies) is the same process as before.
- The pack output moves from the CI log into per-pack log files in the
scratch root. A failure reports the last 2000 characters of the failing
pack's log; a successful pack's output is discarded with the scratch
root.
- The tarball checks (`tar -tzf`) now run after all packs instead of
after each pack. A bundled Codex payload is still rejected before any
tarball reaches the consumer; the only change is that the error arrives
a few seconds later.
- Low risk otherwise. The bundled installs, the consumer install, and
every probe are unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.1 (Anthropic), model ID `claude-fable-5-1`, run as a
Paperclip agent through the Claude Code local adapter with tool use
(shell, file edits, GitHub CLI). Reasoning mode: extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (run 38146108750: `server
(11/12)` failed on its first attempt on a teardown `EACCES` while
removing the per-worker `PAPERCLIP_HOME` after all 74 files and 1257
tests had passed, in a file this PR does not touch; the re-run of that
job passed and every check is green)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-11 12:02:19 -07:00

138 lines
11 KiB
JSON
Raw Blame History

{
"name": "paperclip",
"private": true,
"type": "module",
"scripts": {
"postinstall": "node scripts/link-plugin-dev-sdk.mjs",
"preflight:workspace-links": "node cli/node_modules/tsx/dist/cli.mjs scripts/ensure-workspace-package-links.ts",
"dev": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
"dev:watch": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
"dev:once": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts dev",
"dev:list": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts list",
"dev:stop": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts stop",
"dev:server": "pnpm --filter @paperclipai/server dev",
"dev:ui": "pnpm --filter @paperclipai/ui dev",
"dev:mobile": "pnpm --filter @paperclipai/ui build && pnpm --filter @paperclipai/ui preview",
"dev:both": "node scripts/dev-both.mjs",
"prototype:dot": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-demo.ts",
"prototype:dot:live": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-live.ts",
"storybook": "pnpm --filter @paperclipai/ui storybook",
"build-storybook": "pnpm --filter @paperclipai/ui build-storybook",
"build": "pnpm run preflight:workspace-links && pnpm -r build",
"build:feature-catalog": "node cli/node_modules/tsx/dist/cli.mjs scripts/generate-feature-catalog.ts",
"typecheck": "pnpm run preflight:workspace-links && pnpm -r typecheck",
"typecheck:build-gaps": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && pnpm --filter @paperclipai/server build && node scripts/run-typecheck-build-gaps.mjs",
"test": "pnpm run test:run",
"test:watch": "pnpm run preflight:workspace-links && vitest",
"test:run": "pnpm run preflight:workspace-links && node scripts/run-vitest-stable.mjs",
"test:runner-acceptance": "vitest run --config tests/runner-acceptance/vitest.config.ts",
"test:runner-acceptance:typecheck": "tsc -p tests/runner-acceptance/tsconfig.json",
"test:run:general": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode general",
"test:run:serialized": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode serialized",
"db:generate": "pnpm --filter @paperclipai/db generate",
"db:migrate": "pnpm --filter @paperclipai/db migrate",
"issue-references:backfill": "pnpm run preflight:workspace-links && tsx scripts/backfill-issue-reference-mentions.ts",
"secrets:migrate-inline-env": "tsx scripts/migrate-inline-env-secrets.ts",
"db:backup": "./scripts/backup-db.sh",
"paperclipai": "node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts",
"build:npm": "./scripts/build-npm.sh",
"release": "./scripts/release.sh",
"release:canary": "./scripts/release.sh canary",
"release:stable": "./scripts/release.sh stable",
"release:github": "./scripts/create-github-release.sh",
"release:rollback": "./scripts/rollback-latest.sh",
"release:bootstrap-package": "node scripts/bootstrap-npm-package.mjs",
"check:tokens": "node scripts/check-forbidden-tokens.mjs",
"check:token-gates": "node scripts/check-token-gates.mjs && node scripts/sync-agent-palette-tokens.mjs --check && node scripts/sync-cliplab-character.mjs --check",
"check:node-version": "node scripts/check-node-version-policy.mjs",
"check:no-git-push": "node scripts/check-no-git-push.mjs",
"check:module-boundaries": "node scripts/check-module-boundaries.mjs",
"test:check-no-git-push": "node --test scripts/check-no-git-push.test.mjs",
"test:install-sh-docker": "./scripts/test-install-sh-docker.sh",
"test:hermes-gateway-smoke": "node --test scripts/smoke/hermes-gateway-smoke.test.mjs",
"docs:dev": "cd docs && npx mintlify dev",
"smoke:hermes-gateway-join": "./scripts/smoke/hermes-gateway-join.sh",
"smoke:hermes-gateway-e2e": "./scripts/smoke/hermes-gateway-e2e.sh",
"smoke:openclaw-join": "./scripts/smoke/openclaw-join.sh",
"smoke:openclaw-docker-ui": "./scripts/smoke/openclaw-docker-ui.sh",
"smoke:openclaw-sse-standalone": "./scripts/smoke/openclaw-sse-standalone.sh",
"smoke:mcp-fixtures": "node scripts/smoke/mcp-fixture-harness.mjs",
"smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
"test:e2e": "npx playwright test --config tests/e2e/playwright.config.ts",
"test:e2e:runner": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts",
"test:e2e:runner:image-id": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/daytona-image-content.ts",
"test:e2e:runner:judge-first-task": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/first-task-judge.ts",
"test:e2e:runner:dashboard": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/dashboard-regenerate.ts",
"test:e2e:runner:models:update": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/openrouter-models-update.ts",
"test:e2e:runner:history:publish": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/history-publish.ts",
"test:runner-recovery": "vitest run server/src/services/native-runtime/native-replacement-evidence.test.ts server/src/services/native-runtime/stopped-codex-turn.test.ts server/src/services/native-runtime/native-safe-replacement.test.ts",
"test:slack-connector": "node server/src/services/connectors/slack/evals/run-regressions.mjs",
"test:e2e:browser-context": "playwright test --config tests/e2e/playwright-company-context.config.ts",
"test:e2e:runner:browser-support": "playwright test --config tests/runner-e2e/playwright-support.config.ts",
"test:e2e:runner:unit": "vitest run --config tests/runner-e2e/vitest.config.ts && node --test tests/runner-e2e/native-completion-checks.test.mjs tests/runner-e2e/native-completion-git-source.test.mjs tests/runner-e2e/native-completion-source-contract.test.mjs",
"test:e2e:runner:stock-harness": "node tests/runner-e2e/stock-harness-checks.mjs",
"test:e2e:runner:typecheck": "tsc -p tests/runner-e2e/tsconfig.json",
"test:e2e:runner:report": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/report.ts",
"test:runner-workflow-evals": "pnpm --filter @paperclipai/paperclip-eval-kernel build && pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals",
"test:e2e:mcp-user-stories": "node scripts/e2e-mcp-user-stories.mjs",
"test:e2e:connection-intents": "npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/connection-intents.spec.ts",
"test:e2e:headed": "npx playwright test --config tests/e2e/playwright.config.ts --headed",
"test:e2e:multiuser-authenticated": "npx playwright test --config tests/e2e/playwright-multiuser-authenticated.config.ts",
"evals:smoke": "cd evals/promptfoo && npx promptfoo@0.103.3 eval",
"test:release-smoke": "npx playwright test --config tests/release-smoke/playwright.config.ts",
"test:release-smoke:headed": "npx playwright test --config tests/release-smoke/playwright.config.ts --headed",
"test:canary-onboarding-smoke": "npx playwright test --config tests/canary-onboarding/playwright.config.ts",
"metrics:paperclip-commits": "tsx scripts/paperclip-commit-metrics.ts",
"perf:issue-chat-long-thread": "node scripts/measure-issue-chat-long-thread.mjs",
"connections:ingest-app-definitions": "node scripts/ingest-app-definitions.mjs",
"test:lifecycle-baseline": "node tests/lifecycle-baseline/run.mjs",
"test:lifecycle-baseline:support": "node --test tests/lifecycle-baseline/report.test.mjs",
"test:lifecycle-baseline:typecheck": "tsc -p tests/lifecycle-baseline/tsconfig.json"
},
"devDependencies": {
"@playwright/test": "^1.62.1",
"agentmail": "^0.5.31",
"cross-env": "^10.1.0",
"esbuild": "^0.28.2",
"typescript": "^7.0.2",
"vitest": "^5.0.3"
},
"engines": {
"node": ">=24.11.0"
},
"packageManager": "pnpm@9.15.4",
"pnpm": {
"patchedDependencies": {
"embedded-postgres@18.1.0-beta.16": "patches/embedded-postgres@18.1.0-beta.16.patch",
"acpx@0.12.0": "patches/acpx@0.12.0.patch",
"acpx@0.13.1": "patches/acpx@0.13.1.patch",
"@agentclientprotocol/claude-agent-acp@0.73.0": "patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch",
"@agentclientprotocol/codex-acp@1.6.2": "patches/@agentclientprotocol__codex-acp@1.6.2.patch",
"@chat-adapter/telegram@4.39.0": "patches/@chat-adapter__telegram@4.39.0.patch",
"@chat-adapter/teams@4.39.0": "patches/@chat-adapter__teams@4.39.0.patch",
"@chat-adapter/slack@4.39.0": "patches/@chat-adapter__slack@4.39.0.patch",
"@chat-adapter/discord@4.39.0": "patches/@chat-adapter__discord@4.39.0.patch",
"@chat-adapter/github@4.39.0": "patches/@chat-adapter__github@4.39.0.patch",
"@discordjs/ws@1.2.3": "patches/@discordjs__ws@1.2.3.patch",
"postgres@3.4.9": "patches/postgres@3.4.9.patch"
},
"overrides": {
"@agentclientprotocol/codex-acp@1.6.2>@openai/codex": "0.160.0",
"@agentclientprotocol/claude-agent-acp@0.73.0>@anthropic-ai/claude-agent-sdk": "0.3.286",
"rollup": ">=4.59.0",
"react": "^19.2.8",
"react-dom": "^19.2.8",
"@codemirror/state": "^6.7.2",
"@codemirror/view": "^6.43.11",
"@lezer/common": "^1.5.2"
}
}
}
Reference in new issue View Git Blame Copy Permalink