Files
PaperClipAI/package.json
T
DottaandClaude Opus 5.5 328a954cc1 test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane must run 1,000 agents at once on one workspace
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`)
> - Above about 50 agents, the evidence is a load model only. Nobody has
run 200 or 1,000 real agents
> - Phase 3 of the plan proves the finish line with real runs on hosted
instances, in a fixed order
> - This pull request adds the harness for those runs. It does not run
it against any real instance
> - The benefit is that each proof step is one command with a written
cost estimate, explicit pass/fail criteria, and a guaranteed cleanup

## Linked Issues or Issue Description

**What problem does this solve?**
The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents.
No tool exists that creates real agents and tasks on a hosted instance,
ramps concurrency, measures the control plane, and judges the plan's
finish line.

**Proposed solution**
A script under `scripts/` that ramps real agents through the product
API, stops at the first failed criterion, writes a machine-readable
result, and always cleans up what it created.

**Alternatives considered**
Extend the restart-chaos hosted mode. That mode grades release swaps at
up to 30 concurrent tasks. The ramp reuses its target guard and its
cleanup order, but it has a different workload, different measures and a
different stop rule.

## What Changed

- `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness.
It reads the base URL from a flag or the environment and the credentials
from the environment only. It creates or reuses a company, then creates
agents, tasks, a project and an optional provider secret. It ramps
concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps
each agent on one small deterministic task at a time.
- It measures per step: tasks started, finished, failed and needing
Retry; failed runs; peak running agents; queue wait; time to first token
and first activity (from sampled run events); run duration; board GET
latency. An optional metrics endpoint supplies event-loop, statement and
pool figures. A missing source is reported as "not measured", never as
zero.
- It evaluates the criteria from the plan's finish line plus harness
criteria, writes them into `result.json` and `SUMMARY.md`, and stops at
the first failure.
- Cleanup runs in a `finally`: pause agents, cancel live runs, archive
tasks, terminate and delete agents, delete the stored secret, archive
the project and the created company, and wait for sandbox leases.
`--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run`
prints the plan and the estimate and makes no request.
- Target guard: `--target-kind` is required and must match the host.
Shared staging and production hosts need a second explicit flag. The
script asks for the host name before it starts unless `--yes` is given.
A run above the spend threshold needs `--confirm-spend <max USD>`.
- `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an
in-process fake API with a virtual clock. They cover ramp logic, stop on
failure, criteria evaluation, cleanup after a mid-run crash and an
interrupt, `--cleanup-only` with and without a state file, dry run, the
production-host guard, the spend gate, and that credentials never appear
in output or files.
- `scripts/scale-ramp/tsconfig.json` is added to the test programs that
CI typechecks. The test file joins the `test:release-registry` node test
list, which CI runs.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3
runbook with commands, cost, criteria, proof order, target prerequisites
and open risks.

## Verification

- `node scripts/ci-profile/node-test.mjs
scripts/scale-ramp/scale-ramp.test.ts
scripts/typecheck-test-programs.test.mjs`: 47 tests pass.
- `node node_modules/typescript/bin/tsc -p
scripts/scale-ramp/tsconfig.json`: no errors.
- `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts
--base-url https://scale1.test.paperclip.app --target-kind test-stack
--steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate
and the criteria, and makes no request.
- The harness was not run against any real instance. That needs the
owner's go-ahead.

## Risks

- Low risk for the product: the change adds a script, tests and docs
only. No server code changes.
- A real run creates real agents, tasks and sandboxes and spends money.
The guard, the typed confirmation, the spend gate, the task cap and the
recorded-spend stop limit this. The runbook lists the open risks.
- The cost defaults are assumptions until a first real run records
actual spend.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:10:51 -06:00

141 lines
12 KiB
JSON

{
"name": "paperclip",
"private": true,
"type": "module",
"scripts": {
"perf:workspace-admission": "node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-workspace-admission.ts",
"postinstall": "node scripts/link-plugin-dev-sdk.mjs",
"preflight:workspace-links": "node cli/node_modules/tsx/dist/cli.mjs scripts/ensure-workspace-package-links.ts",
"dev": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
"dev:watch": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
"dev:once": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts dev",
"dev:list": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts list",
"dev:stop": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts stop",
"dev:server": "pnpm --filter @paperclipai/server dev",
"dev:ui": "pnpm --filter @paperclipai/ui dev",
"dev:mobile": "pnpm --filter @paperclipai/ui build && pnpm --filter @paperclipai/ui preview",
"dev:both": "node scripts/dev-both.mjs",
"prototype:dot": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-demo.ts",
"prototype:dot:live": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-live.ts",
"storybook": "pnpm --filter @paperclipai/ui storybook",
"build-storybook": "pnpm --filter @paperclipai/ui build-storybook",
"build": "pnpm run preflight:workspace-links && pnpm -r build",
"build:feature-catalog": "node cli/node_modules/tsx/dist/cli.mjs scripts/generate-feature-catalog.ts",
"typecheck": "pnpm run preflight:workspace-links && pnpm -r typecheck",
"typecheck:build-gaps": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && pnpm --filter @paperclipai/server build && node scripts/run-typecheck-build-gaps.mjs",
"test": "pnpm run test:run",
"test:watch": "pnpm run preflight:workspace-links && vitest",
"test:run": "pnpm run preflight:workspace-links && node scripts/run-vitest-stable.mjs",
"test:runner-acceptance": "vitest run --config tests/runner-acceptance/vitest.config.ts",
"test:runner-acceptance:typecheck": "tsc -p tests/runner-acceptance/tsconfig.json",
"test:run:general": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode general",
"test:run:serialized": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode serialized",
"db:generate": "pnpm --filter @paperclipai/db generate",
"db:migrate": "pnpm --filter @paperclipai/db migrate",
"issue-references:backfill": "pnpm run preflight:workspace-links && tsx scripts/backfill-issue-reference-mentions.ts",
"secrets:migrate-inline-env": "tsx scripts/migrate-inline-env-secrets.ts",
"db:backup": "./scripts/backup-db.sh",
"paperclipai": "node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts",
"build:npm": "./scripts/build-npm.sh",
"release": "./scripts/release.sh",
"release:canary": "./scripts/release.sh canary",
"release:stable": "./scripts/release.sh stable",
"release:github": "./scripts/create-github-release.sh",
"release:rollback": "./scripts/rollback-latest.sh",
"release:bootstrap-package": "node scripts/bootstrap-npm-package.mjs",
"check:tokens": "node scripts/check-forbidden-tokens.mjs",
"check:token-gates": "node scripts/check-token-gates.mjs && node scripts/sync-agent-palette-tokens.mjs --check && node scripts/sync-cliplab-character.mjs --check",
"check:node-version": "node scripts/check-node-version-policy.mjs",
"check:no-git-push": "node scripts/check-no-git-push.mjs",
"check:module-boundaries": "node scripts/check-module-boundaries.mjs",
"test:check-no-git-push": "node --test scripts/check-no-git-push.test.mjs",
"test:install-sh-docker": "./scripts/test-install-sh-docker.sh",
"test:hermes-gateway-smoke": "node --test scripts/smoke/hermes-gateway-smoke.test.mjs",
"docs:dev": "cd docs && npx mintlify dev",
"smoke:hermes-gateway-join": "./scripts/smoke/hermes-gateway-join.sh",
"smoke:hermes-gateway-e2e": "./scripts/smoke/hermes-gateway-e2e.sh",
"smoke:openclaw-join": "./scripts/smoke/openclaw-join.sh",
"smoke:openclaw-docker-ui": "./scripts/smoke/openclaw-docker-ui.sh",
"smoke:openclaw-sse-standalone": "./scripts/smoke/openclaw-sse-standalone.sh",
"smoke:mcp-fixtures": "node scripts/smoke/mcp-fixture-harness.mjs",
"smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/agent-preflight.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/draft-stable-notes.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs scripts/__tests__/ci-typecheck-lane.test.mjs scripts/scale-ramp/scale-ramp.test.ts",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
"test:e2e": "node scripts/ci-profile/run.mjs playwright npx playwright test --config tests/e2e/playwright.config.ts",
"test:e2e:runner": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts",
"test:e2e:runner:image-id": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/daytona-image-content.ts",
"test:e2e:runner:judge-first-task": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/first-task-judge.ts",
"test:e2e:runner:dashboard": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/dashboard-regenerate.ts",
"test:e2e:runner:models:update": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/openrouter-models-update.ts",
"test:e2e:runner:history:publish": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/history-publish.ts",
"test:runner-recovery": "vitest run server/src/services/native-runtime/native-replacement-evidence.test.ts server/src/services/native-runtime/stopped-codex-turn.test.ts server/src/services/native-runtime/native-safe-replacement.test.ts",
"test:slack-connector": "node server/src/services/connectors/slack/evals/run-regressions.mjs",
"test:e2e:browser-context": "playwright test --config tests/e2e/playwright-company-context.config.ts",
"test:e2e:runner:browser-support": "playwright test --config tests/runner-e2e/playwright-support.config.ts",
"test:e2e:runner:unit": "node tests/runner-e2e/run-unit.mjs",
"test:e2e:runner:stock-harness": "node tests/runner-e2e/stock-harness-checks.mjs",
"test:e2e:runner:typecheck": "node scripts/typecheck-test-projects.mjs tests/runner-e2e",
"test:e2e:runner:report": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/report.ts",
"test:runner-workflow-evals": "pnpm --filter @paperclipai/paperclip-eval-kernel build && pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals",
"test:e2e:mcp-user-stories": "node scripts/e2e-mcp-user-stories.mjs",
"test:e2e:connection-intents": "npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/connection-intents.spec.ts",
"test:e2e:headed": "npx playwright test --config tests/e2e/playwright.config.ts --headed",
"test:e2e:multiuser-authenticated": "npx playwright test --config tests/e2e/playwright-multiuser-authenticated.config.ts",
"evals:smoke": "cd evals/promptfoo && npx promptfoo@0.103.3 eval",
"test:release-smoke": "npx playwright test --config tests/release-smoke/playwright.config.ts",
"test:release-smoke:headed": "npx playwright test --config tests/release-smoke/playwright.config.ts --headed",
"test:canary-onboarding-smoke": "npx playwright test --config tests/canary-onboarding/playwright.config.ts",
"metrics:paperclip-commits": "tsx scripts/paperclip-commit-metrics.ts",
"perf:issue-chat-long-thread": "node scripts/measure-issue-chat-long-thread.mjs",
"connections:ingest-app-definitions": "node scripts/ingest-app-definitions.mjs",
"test:lifecycle-baseline": "node tests/lifecycle-baseline/run.mjs",
"test:lifecycle-baseline:support": "node --test tests/lifecycle-baseline/report.test.mjs",
"test:lifecycle-baseline:typecheck": "node scripts/typecheck-test-projects.mjs tests/lifecycle-baseline",
"verify": "node --import ./cli/node_modules/tsx/dist/loader.mjs scripts/ensure-workspace-package-links.ts && node scripts/local-verify.mjs"
},
"devDependencies": {
"@playwright/test": "^1.62.1",
"agentmail": "^0.5.31",
"cross-env": "^10.1.0",
"esbuild": "^0.28.2",
"oxc-parser": "0.127.0",
"typescript": "^7.0.2",
"vitest": "^5.0.3",
"yaml": "^2.8.1",
"turbo": "2.11.10"
},
"engines": {
"node": ">=24.11.0"
},
"packageManager": "pnpm@9.15.4",
"pnpm": {
"patchedDependencies": {
"embedded-postgres@18.1.0-beta.16": "patches/embedded-postgres@18.1.0-beta.16.patch",
"acpx@0.22.0": "patches/acpx@0.22.0.patch",
"@agentclientprotocol/claude-agent-acp@0.73.0": "patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch",
"@agentclientprotocol/codex-acp@1.6.2": "patches/@agentclientprotocol__codex-acp@1.6.2.patch",
"@chat-adapter/telegram@4.39.0": "patches/@chat-adapter__telegram@4.39.0.patch",
"@chat-adapter/teams@4.39.0": "patches/@chat-adapter__teams@4.39.0.patch",
"@chat-adapter/slack@4.39.0": "patches/@chat-adapter__slack@4.39.0.patch",
"@chat-adapter/discord@4.39.0": "patches/@chat-adapter__discord@4.39.0.patch",
"@chat-adapter/github@4.39.0": "patches/@chat-adapter__github@4.39.0.patch",
"@discordjs/ws@1.2.3": "patches/@discordjs__ws@1.2.3.patch",
"postgres@3.4.9": "patches/postgres@3.4.9.patch"
},
"overrides": {
"@agentclientprotocol/codex-acp@1.6.2>@openai/codex": "0.160.0",
"@agentclientprotocol/claude-agent-acp@0.73.0>@anthropic-ai/claude-agent-sdk": "0.3.286",
"rollup": ">=4.59.0",
"react": "^19.2.8",
"react-dom": "^19.2.8",
"@codemirror/state": "^6.7.2",
"@codemirror/view": "^6.43.11",
"@lezer/common": "^1.5.2"
}
}
}