mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI splits the server tests across 12 shards; `ci / verify` fails when any shard fails > - Since 2026-11-16 the server shards were cancelled at their 20-minute job limit with no output (shard 2/12 five times, plus shards 1, 3 and 9), which blocked unrelated pull requests until reruns > - Turbo prints a task's output only when it ends, so a cancelled shard never names the stuck file > - Recomputing each cancelled shard's file list at its own merge commit put `issue-recovery-live-run-agent-census.test.ts` in 12 of 14 of them; it is also the slowest file in passing shards (235-300 s) > - That census started 9-14 real agent runs per file while `setTimeout` was faked, waited on them with no bound, and had 30-minute test timeouts, so a stuck wait could never fail inside the job > - This pull request stops the census from starting real runs, bounds every wait, and adds a CI reporter that fails a run naming any file that stops making progress > - The benefit is that this file cannot hang a shard, and the next stuck file fails in minutes with its name and step in the log ## Linked Issues or Issue Description Refs #21867 (the same fix for the sibling agent-row census), #21831 (the behavior this census guards) **What happened?** `ci / General tests (server (2/12))` was cancelled at 20m16s in actions runs 43865258195, 43926799428, 43932238377, 43934278368, 43944290677 and 43935722497 (both attempts), and other server shards hit the same limit (for example 43901003914 shard 1, 43945768035 shard 3, 43845343831 shard 9). The hung jobs show CPU near 0 for the last 9 minutes and printed nothing after `Running //#test:run:general`. Shard membership changes with every merge commit. To compare the shards, I recomputed each job's file list at its exact merge commit with `GITHUB_WORKFLOW=PR` (the PR workflow removes the native-runner suites from these shards, so a local dry run without it gives a different list). `server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` was in 12 of the 14 hung shards. Why that file: - During its 10-minute fake-timer warm-up, the "live" task was `in_progress` with no run. That is real assigned work, so recovery woke its agent and a real `process`-adapter run started while `setTimeout` was faked. The run lost its lease ("Legacy controller lease lost"), recovery retried, and a local run of the file showed 13 real run executions. - No wait in the file had a bound, and each test had a 30-minute timeout, so a delivery pass waiting on a faked timer or a run holding a lock never failed inside the 20-minute job. - Under local CPU contention, 3 of 4 copies of the old file failed the census assertion (39 settled checks, caused by the stray runs), and each copy took about 270 s with 9-14 real runs. Fixing the runs exposed a second problem in the fixture: the seeded tasks were stamped "now". The first 5-minute backstop after the live run's first hint re-checks rows changed within 5 minutes before the last full pass, so it re-checked all 860 seeded tasks. The old file passed only because the stray runs happened to trigger that backstop during the warm-up. **Expected behavior** The census never starts a real run, every wait has a bound that names what it waited for, and a CI test file that stops making progress fails the job quickly with its name. ## What Changed - `issue-recovery-live-run-agent-census.test.ts` - The live task is a settled review until its run starts; `startLiveRun` now writes the checkout (`in_progress` + `executionRunId`) the way execution start does. - Population agents and the held agent take no on-demand wakes (`wakeOnDemand: false`), so a recovery check cannot dispatch a run. The census counts queue checks, so this does not change what it measures. - The seeded population is backdated 2 hours ("settled long before the window", as the file's comment already said). - Each step (seed, worker start, run start, run-event tick, fake-timer advance, pass settlement, direct writes) has a real 20-second bound captured before the timers are faked, with a message naming the step. Test timeout is 4 minutes instead of 30. - New `scripts/ci/stalled-file-reporter.mjs`, enabled in GitHub Actions from the root `vitest.config.ts`: when a running file starts no test or hook and reports no result for 8 minutes (`PAPERCLIP_VITEST_FILE_STALL_MS` overrides), it prints a `::error file=...` annotation naming the file and the step it was in, cancels the run and exits 1. The slowest healthy server file runs about 5 minutes in total, so 8 minutes without any progress means a stuck file. Tests: `scripts/ci/stalled-file-reporter.test.mjs` (added to `test:release-registry`). ## Verification - Census, red: the old file runs 13 real agent runs locally (`heartbeat execution setup failed`, `LegacyControllerLeaseLostError`); with 4 copies under CPU contention, 3 failed with 39 settled checks at about 270 s each. - Census, green: 4/4 tests pass, 0 real runs, about 80 s locally. - Census still guards #21831: with the `ag.status <> 'running'` line removed from `changed-tasks.ts`, the first test fails with 534 settled checks and 540 backstop checks, the numbers #21831 recorded. - Reporter, red/green: the real-Vitest case (a file awaiting a promise that never settles, with a 10-minute test timeout) is still running when the 20-second test bound ends without the reporter; with the reporter it exits 1 in about 8 s and prints the annotation naming the file and the test. - Reporter on a real server file: `PAPERCLIP_VITEST_FILE_STALL_MS=3000` on `issue-recovery-recorded-decision-idle.test.ts` prints the annotation naming the file and step and exits 1. - `node --test scripts/ci/stalled-file-reporter.test.mjs`: 5/5 pass. ## Risks - Test and CI-tooling change only. No product code changed. - The census's agents no longer take on-demand wakes. The census measures recovery queue checks, not dispatch, and still fails without the #21831 fix. - The reporter stops the whole CI run when one file stalls for 8 minutes. A legitimate single test or hook that runs more than 8 minutes with no other progress in its file would now fail; none exists today (the slowest whole file is about 5 minutes). The limit is configurable. - Other fake-timer recovery files still have long test timeouts and unbounded pass waits (for example `issue-recovery-activity-hints`, 30 minutes). The reporter now caps any of them at 8 minutes without progress and names the file; bounding them is follow-up work. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
141 lines
12 KiB
JSON
141 lines
12 KiB
JSON
{
|
|
"name": "paperclip",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"perf:workspace-admission": "node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-workspace-admission.ts",
|
|
"postinstall": "node scripts/link-plugin-dev-sdk.mjs",
|
|
"preflight:workspace-links": "node cli/node_modules/tsx/dist/cli.mjs scripts/ensure-workspace-package-links.ts",
|
|
"dev": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
|
|
"dev:watch": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts watch",
|
|
"dev:once": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-runner.ts dev",
|
|
"dev:list": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts list",
|
|
"dev:stop": "pnpm --filter @paperclipai/server exec tsx ../scripts/dev-service.ts stop",
|
|
"dev:server": "pnpm --filter @paperclipai/server dev",
|
|
"dev:ui": "pnpm --filter @paperclipai/ui dev",
|
|
"dev:mobile": "pnpm --filter @paperclipai/ui build && pnpm --filter @paperclipai/ui preview",
|
|
"dev:both": "node scripts/dev-both.mjs",
|
|
"prototype:dot": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-demo.ts",
|
|
"prototype:dot:live": "pnpm --filter @paperclipai/server exec tsx scripts/dot-runner-live.ts",
|
|
"storybook": "pnpm --filter @paperclipai/ui storybook",
|
|
"build-storybook": "pnpm --filter @paperclipai/ui build-storybook",
|
|
"build": "pnpm run preflight:workspace-links && pnpm -r build",
|
|
"build:feature-catalog": "node cli/node_modules/tsx/dist/cli.mjs scripts/generate-feature-catalog.ts",
|
|
"typecheck": "pnpm run preflight:workspace-links && pnpm -r typecheck",
|
|
"typecheck:build-gaps": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && pnpm --filter @paperclipai/server build && node scripts/run-typecheck-build-gaps.mjs",
|
|
"test": "pnpm run test:run",
|
|
"test:watch": "pnpm run preflight:workspace-links && vitest",
|
|
"test:run": "pnpm run preflight:workspace-links && node scripts/run-vitest-stable.mjs",
|
|
"test:runner-acceptance": "vitest run --config tests/runner-acceptance/vitest.config.ts",
|
|
"test:runner-acceptance:typecheck": "tsc -p tests/runner-acceptance/tsconfig.json",
|
|
"test:run:general": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode general",
|
|
"test:run:serialized": "pnpm run preflight:workspace-links && pnpm --filter @paperclipai/plugin-sdk ensure-build-deps && node scripts/run-vitest-stable.mjs --mode serialized",
|
|
"db:generate": "pnpm --filter @paperclipai/db generate",
|
|
"db:migrate": "pnpm --filter @paperclipai/db migrate",
|
|
"issue-references:backfill": "pnpm run preflight:workspace-links && tsx scripts/backfill-issue-reference-mentions.ts",
|
|
"secrets:migrate-inline-env": "tsx scripts/migrate-inline-env-secrets.ts",
|
|
"db:backup": "./scripts/backup-db.sh",
|
|
"paperclipai": "node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts",
|
|
"build:npm": "./scripts/build-npm.sh",
|
|
"release": "./scripts/release.sh",
|
|
"release:canary": "./scripts/release.sh canary",
|
|
"release:stable": "./scripts/release.sh stable",
|
|
"release:github": "./scripts/create-github-release.sh",
|
|
"release:rollback": "./scripts/rollback-latest.sh",
|
|
"release:bootstrap-package": "node scripts/bootstrap-npm-package.mjs",
|
|
"check:tokens": "node scripts/check-forbidden-tokens.mjs",
|
|
"check:token-gates": "node scripts/check-token-gates.mjs && node scripts/sync-agent-palette-tokens.mjs --check && node scripts/sync-cliplab-character.mjs --check",
|
|
"check:node-version": "node scripts/check-node-version-policy.mjs",
|
|
"check:no-git-push": "node scripts/check-no-git-push.mjs",
|
|
"check:module-boundaries": "node scripts/check-module-boundaries.mjs",
|
|
"test:check-no-git-push": "node --test scripts/check-no-git-push.test.mjs",
|
|
"test:install-sh-docker": "./scripts/test-install-sh-docker.sh",
|
|
"test:hermes-gateway-smoke": "node --test scripts/smoke/hermes-gateway-smoke.test.mjs",
|
|
"docs:dev": "cd docs && npx mintlify dev",
|
|
"smoke:hermes-gateway-join": "./scripts/smoke/hermes-gateway-join.sh",
|
|
"smoke:hermes-gateway-e2e": "./scripts/smoke/hermes-gateway-e2e.sh",
|
|
"smoke:openclaw-join": "./scripts/smoke/openclaw-join.sh",
|
|
"smoke:openclaw-docker-ui": "./scripts/smoke/openclaw-docker-ui.sh",
|
|
"smoke:openclaw-sse-standalone": "./scripts/smoke/openclaw-sse-standalone.sh",
|
|
"smoke:mcp-fixtures": "node scripts/smoke/mcp-fixture-harness.mjs",
|
|
"smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
|
|
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
|
|
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
|
|
"test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/ci/retry-reporter.test.mjs scripts/ci/stalled-file-reporter.test.mjs scripts/__tests__/runner-vitest-package-lanes.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/agent-build-cache.test.mjs scripts/__tests__/agent-preflight.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/draft-stable-notes.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs scripts/__tests__/ci-typecheck-lane.test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/scale-ramp/scale-ramp-spend.test.ts",
|
|
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
|
|
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
|
|
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
|
|
"test:e2e": "node scripts/ci-profile/run.mjs playwright npx playwright test --config tests/e2e/playwright.config.ts",
|
|
"test:e2e:runner": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts",
|
|
"test:e2e:runner:image-id": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/daytona-image-content.ts",
|
|
"test:e2e:runner:judge-first-task": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/first-task-judge.ts",
|
|
"test:e2e:runner:dashboard": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/dashboard-regenerate.ts",
|
|
"test:e2e:runner:models:update": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/openrouter-models-update.ts",
|
|
"test:e2e:runner:history:publish": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/history-publish.ts",
|
|
"test:runner-recovery": "vitest run server/src/services/native-runtime/native-replacement-evidence.test.ts server/src/services/native-runtime/stopped-codex-turn.test.ts server/src/services/native-runtime/native-safe-replacement.test.ts",
|
|
"test:slack-connector": "node server/src/services/connectors/slack/evals/run-regressions.mjs",
|
|
"test:e2e:browser-context": "playwright test --config tests/e2e/playwright-company-context.config.ts",
|
|
"test:e2e:runner:browser-support": "playwright test --config tests/runner-e2e/playwright-support.config.ts",
|
|
"test:e2e:runner:unit": "node tests/runner-e2e/run-unit.mjs",
|
|
"test:e2e:runner:stock-harness": "node tests/runner-e2e/stock-harness-checks.mjs",
|
|
"test:e2e:runner:typecheck": "node scripts/typecheck-test-projects.mjs tests/runner-e2e",
|
|
"test:e2e:runner:report": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/report.ts",
|
|
"test:runner-workflow-evals": "pnpm --filter @paperclipai/paperclip-eval-kernel build && pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals",
|
|
"test:e2e:mcp-user-stories": "node scripts/e2e-mcp-user-stories.mjs",
|
|
"test:e2e:connection-intents": "npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/connection-intents.spec.ts",
|
|
"test:e2e:headed": "npx playwright test --config tests/e2e/playwright.config.ts --headed",
|
|
"test:e2e:multiuser-authenticated": "npx playwright test --config tests/e2e/playwright-multiuser-authenticated.config.ts",
|
|
"evals:smoke": "cd evals/promptfoo && npx promptfoo@0.103.3 eval",
|
|
"test:release-smoke": "npx playwright test --config tests/release-smoke/playwright.config.ts",
|
|
"test:release-smoke:headed": "npx playwright test --config tests/release-smoke/playwright.config.ts --headed",
|
|
"test:canary-onboarding-smoke": "npx playwright test --config tests/canary-onboarding/playwright.config.ts",
|
|
"metrics:paperclip-commits": "tsx scripts/paperclip-commit-metrics.ts",
|
|
"perf:issue-chat-long-thread": "node scripts/measure-issue-chat-long-thread.mjs",
|
|
"connections:ingest-app-definitions": "node scripts/ingest-app-definitions.mjs",
|
|
"test:lifecycle-baseline": "node tests/lifecycle-baseline/run.mjs",
|
|
"test:lifecycle-baseline:support": "node --test tests/lifecycle-baseline/report.test.mjs",
|
|
"test:lifecycle-baseline:typecheck": "node scripts/typecheck-test-projects.mjs tests/lifecycle-baseline",
|
|
"verify": "node --import ./cli/node_modules/tsx/dist/loader.mjs scripts/ensure-workspace-package-links.ts && node scripts/local-verify.mjs"
|
|
},
|
|
"devDependencies": {
|
|
"@playwright/test": "^1.62.1",
|
|
"agentmail": "^0.5.31",
|
|
"cross-env": "^10.1.0",
|
|
"esbuild": "^0.28.2",
|
|
"oxc-parser": "0.127.0",
|
|
"typescript": "^7.0.2",
|
|
"vitest": "^5.0.3",
|
|
"yaml": "^2.8.1",
|
|
"turbo": "2.11.10"
|
|
},
|
|
"engines": {
|
|
"node": ">=24.11.0"
|
|
},
|
|
"packageManager": "pnpm@9.15.4",
|
|
"pnpm": {
|
|
"patchedDependencies": {
|
|
"embedded-postgres@18.1.0-beta.16": "patches/embedded-postgres@18.1.0-beta.16.patch",
|
|
"acpx@0.22.0": "patches/acpx@0.22.0.patch",
|
|
"@agentclientprotocol/claude-agent-acp@0.73.0": "patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch",
|
|
"@agentclientprotocol/codex-acp@1.6.2": "patches/@agentclientprotocol__codex-acp@1.6.2.patch",
|
|
"@chat-adapter/telegram@4.39.0": "patches/@chat-adapter__telegram@4.39.0.patch",
|
|
"@chat-adapter/teams@4.39.0": "patches/@chat-adapter__teams@4.39.0.patch",
|
|
"@chat-adapter/slack@4.39.0": "patches/@chat-adapter__slack@4.39.0.patch",
|
|
"@chat-adapter/discord@4.39.0": "patches/@chat-adapter__discord@4.39.0.patch",
|
|
"@chat-adapter/github@4.39.0": "patches/@chat-adapter__github@4.39.0.patch",
|
|
"@discordjs/ws@1.2.3": "patches/@discordjs__ws@1.2.3.patch",
|
|
"postgres@3.4.9": "patches/postgres@3.4.9.patch"
|
|
},
|
|
"overrides": {
|
|
"@agentclientprotocol/codex-acp@1.6.2>@openai/codex": "0.160.0",
|
|
"@agentclientprotocol/claude-agent-acp@0.73.0>@anthropic-ai/claude-agent-sdk": "0.3.286",
|
|
"rollup": ">=4.59.0",
|
|
"react": "^19.2.8",
|
|
"react-dom": "^19.2.8",
|
|
"@codemirror/state": "^6.7.2",
|
|
"@codemirror/view": "^6.43.11",
|
|
"@lezer/common": "^1.5.2"
|
|
}
|
|
}
|
|
}
|