mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling effort uses a real-agent ramp harness (`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted test instance through the product API > - Its first run with agents that really ran (10 agents, 105 tasks, 106 runs) stopped on one failed run, recorded $0 model spend, and left cleanup errors > - Read-only inspection of the instance showed three harness faults. The failed run had a same-run bootstrap retry scheduled 33 s later, but the harness stopped and cleaned up first. Run rows carry no cost, but cost records do (63.7 cents for the run's agents). Terminated agents leave the company agent list, so the harness treated them as done and sent deletes that Paperclip refused with 409 > - These faults make the ramp stop too early, make its spend cap inert, and make cleanup report the wrong cause > - This pull request keeps the ramp going through runs that Paperclip recovers by itself, reads spend from cost records, records board latency per route, and makes cleanup wait for termination and name what it left behind > - The benefit is that the next ramp grades the product, not the harness, and its spend cap works ## Linked Issues or Issue Description Refs #21695 Refs #21757 **What happened?** The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`. The run failed with `runner_ingress_bind_conflict` before its provider session opened. Its run events show `same-run resume persisted` with `recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped within 9 s and paused the agents, so the retry was blocked (`native_retry_admission_ineligible`). The result showed `Recorded model spend: $0`. The harness read `costUsd` from run rows, which is always null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over 104 cost events for the 10 agents. Cleanup reported two `DELETE /api/agents/:id returned 409: Complete termination before deleting the agent`. Those agents were still `lifecycleState: terminating`; the harness waited on the company agent list, which hides terminated agents, so it treated them as done. A later `--cleanup-only` changed nothing because it also found agents only through that list. The board API p90 failed (765 ms), but the result had no per-route timings to say which request was slow. **Expected behavior** A failed run whose task finishes by itself is counted, not a stop. A failed run whose task does not finish within a grace stops the ramp. The share of failed runs is graded at the step end. Spend comes from Paperclip's cost records. Cleanup deletes an agent only after termination completes, and names the agents and sandbox leases still pending. **Steps to reproduce** 1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`. 2. 16 tests fail, among them every new test (recovered run, unrecovered run, spend from cost records, spend cap, board per route, termination wait, still-terminating agents and `--cleanup-only`, leases released during the wait). 3. With this change, all 60 pass. **Paperclip version or commit** Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757. ## What Changed - `criteria.ts`: replace `failed_runs_zero` with `failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the step end). Step metrics add `runsEnded`, `runsFailedRecovered`, `runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`. - `ramp.ts`: a run counts as failed once it was seen `failed` or `timed_out`. It is recovered when its task reaches `done`. It is unrecovered when the task rests not done, needs Retry, or has no live run for `--recovery-grace-minutes` after the failure. A task whose interruption has an automatic retry scheduled (`state: retry_scheduled`, `nextRetryAt` or `retryWait`) does not count as needing Retry. Spend comes from `costs/by-agent` for the run's agents on each poll, and each step reports the spend in its window. Board probes keep a distribution per route. - `config.ts`: add `--max-failed-run-percent` (default 2) and `--recovery-grace-minutes` (default 15). - `cleanup.ts`: read agents by id as well as from the list, so terminated agents in the state file are found. Wait for `lifecycleState` `terminated` or `rejected` by id before a delete. Report `agentsStillTerminating`, `activeSandboxLeaseIds` and `sandboxLeasesReleasedDuringWait`, and name each lease still active with its run and acquire time. - `report.ts`: the summary shows recovered, unrecovered and pending failed runs, board latency per route, and each cleanup error. - `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns cents per agent. Optional failure of the n-th run, a scheduled recovery run, a termination delay (terminated agents leave the list and a delete answers 409 until termination completes), and leases that stay active after their run. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook describes the failed-run criteria, the spend source and the cleanup behavior. ## Verification - `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass. - Red check: with the new fake and tests but the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail, including all 8 new ones. - `tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - Evidence from read-only GETs on the test instance: run `f1892d43` events (bind conflict, scheduled same-run retry, blocked after the pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71 cents for the 10 agents), agent rows by id (`terminating` while absent from the list), lease rows. - Not run: a new real ramp (needs the owner's go-ahead). ## Risks - Low risk. The change touches only the scale-ramp harness, its fake, its tests and one runbook section. No product code changes. - A failed run no longer stops the ramp at once. An unrecovered one stops it after the grace (15 min by default), so a broken step can run up to that long. The step-end failed-run rate still fails a step with more than 2% failed runs. - Spend now needs cost read access for the board token. If `costs/by-agent` cannot be read, the result reports spend as not measured and the spend cap does not trigger. - Cleanup can now take up to 5 more minutes while it waits for termination by id. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only HTTP GETs to the test instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
141 lines
12 KiB
JSON
141 lines
12 KiB
JSON
{
|
|
"name": "paperclip",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"perf:workspace-admission": "node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-workspace-admission.ts",
|
|
"postinstall": "node scripts/link-plugin-dev-sdk.mjs",
|
|
"preflight:workspace-links": "node cli/node_modules/tsx/dist/cli.mjs scripts/ensure-workspace-package-links.ts",
|
|
"dev": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
|
|
"dev:watch": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
|
|
"dev:once": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts dev",
|
|
"dev:list": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts list",
|
|
"dev:stop": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts stop",
|
|
"dev:server": "pnpm --filter @paperclipai/server dev",
|
|
"dev:ui": "pnpm --filter @paperclipai/ui dev",
|
|
"dev:mobile": "pnpm --filter @paperclipai/ui build && pnpm --filter @paperclipai/ui preview",
|
|
"dev:both": "node scripts/dev-both.mjs",
|
|
"prototype:dot": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-demo.ts",
|
|
"prototype:dot:live": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-live.ts",
|
|
"storybook": "pnpm --filter @paperclipai/ui storybook",
|
|
"build-storybook": "pnpm --filter @paperclipai/ui build-storybook",
|
|
"build": "pnpm run preflight:workspace-links && pnpm -r build",
|
|
"build:feature-catalog": "node cli/node_modules/tsx/dist/cli.mjs scripts/generate-feature-catalog.ts",
|
|
"typecheck": "pnpm run preflight:workspace-links && pnpm -r typecheck",
|
|
"typecheck:build-gaps": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && pnpm --filter @paperclipai/server build && node scripts/run-typecheck-build-gaps.mjs",
|
|
"test": "pnpm run test:run",
|
|
"test:watch": "pnpm run preflight:workspace-links && vitest",
|
|
"test:run": "pnpm run preflight:workspace-links && node scripts/run-vitest-stable.mjs",
|
|
"test:runner-acceptance": "vitest run --config tests/runner-acceptance/vitest.config.ts",
|
|
"test:runner-acceptance:typecheck": "tsc -p tests/runner-acceptance/tsconfig.json",
|
|
"test:run:general": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode general",
|
|
"test:run:serialized": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode serialized",
|
|
"db:generate": "pnpm --filter @paperclipai/db generate",
|
|
"db:migrate": "pnpm --filter @paperclipai/db migrate",
|
|
"issue-references:backfill": "pnpm run preflight:workspace-links && tsx scripts/backfill-issue-reference-mentions.ts",
|
|
"secrets:migrate-inline-env": "tsx scripts/migrate-inline-env-secrets.ts",
|
|
"db:backup": "./scripts/backup-db.sh",
|
|
"paperclipai": "node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts",
|
|
"build:npm": "./scripts/build-npm.sh",
|
|
"release": "./scripts/release.sh",
|
|
"release:canary": "./scripts/release.sh canary",
|
|
"release:stable": "./scripts/release.sh stable",
|
|
"release:github": "./scripts/create-github-release.sh",
|
|
"release:rollback": "./scripts/rollback-latest.sh",
|
|
"release:bootstrap-package": "node scripts/bootstrap-npm-package.mjs",
|
|
"check:tokens": "node scripts/check-forbidden-tokens.mjs",
|
|
"check:token-gates": "node scripts/check-token-gates.mjs && node scripts/sync-agent-palette-tokens.mjs --check && node scripts/sync-cliplab-character.mjs --check",
|
|
"check:node-version": "node scripts/check-node-version-policy.mjs",
|
|
"check:no-git-push": "node scripts/check-no-git-push.mjs",
|
|
"check:module-boundaries": "node scripts/check-module-boundaries.mjs",
|
|
"test:check-no-git-push": "node --test scripts/check-no-git-push.test.mjs",
|
|
"test:install-sh-docker": "./scripts/test-install-sh-docker.sh",
|
|
"test:hermes-gateway-smoke": "node --test scripts/smoke/hermes-gateway-smoke.test.mjs",
|
|
"docs:dev": "cd docs && npx mintlify dev",
|
|
"smoke:hermes-gateway-join": "./scripts/smoke/hermes-gateway-join.sh",
|
|
"smoke:hermes-gateway-e2e": "./scripts/smoke/hermes-gateway-e2e.sh",
|
|
"smoke:openclaw-join": "./scripts/smoke/openclaw-join.sh",
|
|
"smoke:openclaw-docker-ui": "./scripts/smoke/openclaw-docker-ui.sh",
|
|
"smoke:openclaw-sse-standalone": "./scripts/smoke/openclaw-sse-standalone.sh",
|
|
"smoke:mcp-fixtures": "node scripts/smoke/mcp-fixture-harness.mjs",
|
|
"smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
|
|
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
|
|
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
|
|
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/agent-preflight.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/draft-stable-notes.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs scripts/__tests__/ci-typecheck-lane.test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/scale-ramp/scale-ramp-spend.test.ts",
|
|
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
|
|
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
|
|
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
|
|
"test:e2e": "node scripts/ci-profile/run.mjs playwright npx playwright test --config tests/e2e/playwright.config.ts",
|
|
"test:e2e:runner": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts",
|
|
"test:e2e:runner:image-id": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/daytona-image-content.ts",
|
|
"test:e2e:runner:judge-first-task": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/first-task-judge.ts",
|
|
"test:e2e:runner:dashboard": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/dashboard-regenerate.ts",
|
|
"test:e2e:runner:models:update": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/openrouter-models-update.ts",
|
|
"test:e2e:runner:history:publish": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/history-publish.ts",
|
|
"test:runner-recovery": "vitest run server/src/services/native-runtime/native-replacement-evidence.test.ts server/src/services/native-runtime/stopped-codex-turn.test.ts server/src/services/native-runtime/native-safe-replacement.test.ts",
|
|
"test:slack-connector": "node server/src/services/connectors/slack/evals/run-regressions.mjs",
|
|
"test:e2e:browser-context": "playwright test --config tests/e2e/playwright-company-context.config.ts",
|
|
"test:e2e:runner:browser-support": "playwright test --config tests/runner-e2e/playwright-support.config.ts",
|
|
"test:e2e:runner:unit": "node tests/runner-e2e/run-unit.mjs",
|
|
"test:e2e:runner:stock-harness": "node tests/runner-e2e/stock-harness-checks.mjs",
|
|
"test:e2e:runner:typecheck": "node scripts/typecheck-test-projects.mjs tests/runner-e2e",
|
|
"test:e2e:runner:report": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/report.ts",
|
|
"test:runner-workflow-evals": "pnpm --filter @paperclipai/paperclip-eval-kernel build && pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals",
|
|
"test:e2e:mcp-user-stories": "node scripts/e2e-mcp-user-stories.mjs",
|
|
"test:e2e:connection-intents": "npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/connection-intents.spec.ts",
|
|
"test:e2e:headed": "npx playwright test --config tests/e2e/playwright.config.ts --headed",
|
|
"test:e2e:multiuser-authenticated": "npx playwright test --config tests/e2e/playwright-multiuser-authenticated.config.ts",
|
|
"evals:smoke": "cd evals/promptfoo && npx promptfoo@0.103.3 eval",
|
|
"test:release-smoke": "npx playwright test --config tests/release-smoke/playwright.config.ts",
|
|
"test:release-smoke:headed": "npx playwright test --config tests/release-smoke/playwright.config.ts --headed",
|
|
"test:canary-onboarding-smoke": "npx playwright test --config tests/canary-onboarding/playwright.config.ts",
|
|
"metrics:paperclip-commits": "tsx scripts/paperclip-commit-metrics.ts",
|
|
"perf:issue-chat-long-thread": "node scripts/measure-issue-chat-long-thread.mjs",
|
|
"connections:ingest-app-definitions": "node scripts/ingest-app-definitions.mjs",
|
|
"test:lifecycle-baseline": "node tests/lifecycle-baseline/run.mjs",
|
|
"test:lifecycle-baseline:support": "node --test tests/lifecycle-baseline/report.test.mjs",
|
|
"test:lifecycle-baseline:typecheck": "node scripts/typecheck-test-projects.mjs tests/lifecycle-baseline",
|
|
"verify": "node --import ./cli/node_modules/tsx/dist/loader.mjs scripts/ensure-workspace-package-links.ts && node scripts/local-verify.mjs"
|
|
},
|
|
"devDependencies": {
|
|
"@playwright/test": "^1.62.1",
|
|
"agentmail": "^0.5.31",
|
|
"cross-env": "^10.1.0",
|
|
"esbuild": "^0.28.2",
|
|
"oxc-parser": "0.127.0",
|
|
"typescript": "^7.0.2",
|
|
"vitest": "^5.0.3",
|
|
"yaml": "^2.8.1",
|
|
"turbo": "2.11.10"
|
|
},
|
|
"engines": {
|
|
"node": ">=24.11.0"
|
|
},
|
|
"packageManager": "pnpm@9.15.4",
|
|
"pnpm": {
|
|
"patchedDependencies": {
|
|
"embedded-postgres@18.1.0-beta.16": "patches/embedded-postgres@18.1.0-beta.16.patch",
|
|
"acpx@0.22.0": "patches/acpx@0.22.0.patch",
|
|
"@agentclientprotocol/claude-agent-acp@0.73.0": "patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch",
|
|
"@agentclientprotocol/codex-acp@1.6.2": "patches/@agentclientprotocol__codex-acp@1.6.2.patch",
|
|
"@chat-adapter/telegram@4.39.0": "patches/@chat-adapter__telegram@4.39.0.patch",
|
|
"@chat-adapter/teams@4.39.0": "patches/@chat-adapter__teams@4.39.0.patch",
|
|
"@chat-adapter/slack@4.39.0": "patches/@chat-adapter__slack@4.39.0.patch",
|
|
"@chat-adapter/discord@4.39.0": "patches/@chat-adapter__discord@4.39.0.patch",
|
|
"@chat-adapter/github@4.39.0": "patches/@chat-adapter__github@4.39.0.patch",
|
|
"@discordjs/ws@1.2.3": "patches/@discordjs__ws@1.2.3.patch",
|
|
"postgres@3.4.9": "patches/postgres@3.4.9.patch"
|
|
},
|
|
"overrides": {
|
|
"@agentclientprotocol/codex-acp@1.6.2>@openai/codex": "0.160.0",
|
|
"@agentclientprotocol/claude-agent-acp@0.73.0>@anthropic-ai/claude-agent-sdk": "0.3.286",
|
|
"rollup": ">=4.59.0",
|
|
"react": "^19.2.8",
|
|
"react-dom": "^19.2.8",
|
|
"@codemirror/state": "^6.7.2",
|
|
"@codemirror/view": "^6.43.11",
|
|
"@lezer/common": "^1.5.2"
|
|
}
|
|
}
|
|
}
|