mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
bed5f70278
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A Core image is published only after the release verification of the same commit passes (`cloud-readiness.yml` calls `release-verify.yml`) > - The image is ready about 3 minutes before that gate opens, so the slowest verification job sets the staging release time > - The slowest job is a general-server test shard: 11.8-12.7 min, while other server shards take 8-9 min > - The shards are balanced by a duration manifest, but the manifest has not changed since 2026-10-31. The weekly refresh that updates it failed on 2026-11-02 and 2026-11-09 > - About 40 newer suites per shard use the 1 s median fallback weight, but several of them run for 90-200 s, so they pile up on one shard > - This pull request repairs the weekly refresh and refreshes the server weights from three current master verifications > - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8 min of test time) on every master verification and every PR run ## Linked Issues or Issue Description **What happened?** The Core image publication waits about 13.5 min for source verification. The critical path is one general-server shard. In three master verifications (runs 43133035381, 43134251278, 43145252393) the slowest server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and 8.4 min. **Root cause** `scripts/general-server-shard-durations.json` came from one run on 2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice: - 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A server shard log is 30-65 MB. One download took 59 s locally. Three parallel downloads overran the 120 s `execFile` timeout. - 2026-11-09 (run 42580964686): the refresh's own check failed. Merged-PR sources for the Runner lane have a leading `pull` key, and the test accepted only `run, attempt, sha, url`. **Expected behavior** The weekly refresh finishes and keeps the shards balanced as suites are added. ## What Changed - `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a 512 MB buffer. JSON calls keep 120 s. A failed call now gives a short error and does not print the partial 60 MB log. - `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`. It accepts the optional leading `pull` key that `collectMergedPrSamples` writes. The checked-in Runner manifest test and the collector test both use it, so the collector format stays covered. - `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 -> 45 min (the server lanes alone fetch about 30 large logs). - `scripts/general-server-shard-durations.json` and `scripts/serialized-shard-durations.json`: refreshed with `node scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381 43134251278 43145252393` (median of three successful protected-master Cloud readiness runs; 1757/1815 general files and 173/173 serialized files measured). - No test is added, removed, skipped or reassigned to a different lane. No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check are not touched. ## Verification Measured job timings (minutes from the first job start; three master verifications): | Job | 43133035381 | 43134251278 | 43145252393 | | --- | --- | --- | --- | | Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7 | | Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 | | Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 | | Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 | | Chat (slowest of 3) | 8.8 | 10.5 | 9.4 | | Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 | | Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 | | Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 | - Queue to runner start is 0.5-1.1 min for every job. In a server shard, setup (checkout, pnpm, Node, install) is about 50 s and the preflight plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8 min). Setup is not worth caching. - Shard prediction, using the refreshed medians as the cost of each file and the 3 fork workers per shard: - general-server, 10 shards: old weights max 10.3 / min 8.1 min; new weights 8.8 / 8.8 min. - serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights 6.6 / 6.6 min. - The old-weight prediction matches what CI saw: predicted 10.3 and 9.9 min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in run 43133035381. - Red/green: - `node --test --test-name-pattern="gh api" scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix (`log timeout 120000`) and passes after it. - With a `pull` key in the Runner manifest sources, the old "package lane manifest" test failed with the same assertion as run 42580964686. The new test passes. - The refresh script, which died before the fix, completed locally in 8 min after it. - `node --test scripts/ci/test-selection-timings.test.mjs scripts/ci/test-selection-pr-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the complete, non-overlapping shard cover checks. - `actionlint .github/workflows/ci-test-durations.yml`: clean. - Not run: the release-verify workflow itself, the scheduled refresh workflow, and the Runner/e2e lane refresh. The shard times above are predictions from logged per-file timings, not a measured CI run on this branch. ## Risks - Low risk. Weights only move files between shards of the same lane. The shard tests prove every file still runs exactly once. - The predicted gain is about 1.5 min on the slowest server shard (12-12.7 -> about 10.5 min job time). The gate then waits for "Pre-release Runner chaos evals / Restart, replay, trace, and recovery faults" (10.2-11.9 min). So the expected gate saving is about 1 min (13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a shard is about ±1 min. - A larger log timeout can make a stuck download wait up to 15 min. The 45 min job timeout still bounds the run. Candidates for the owner (not changed here): - The chaos-eval job is now the bound. Its "server finalization" step takes 7-9 min. Decide if it must gate image publication, or split it. - Server shard logs are up to 811k lines, mostly repeated stack traces from `server/src/services/execution-preparation-timing.ts`. That slows log upload and this refresh. - 9 test files run over 180 s (largest: `chat-channels.integration.test.ts` 429 s, `heartbeat-process-recovery.test.ts` 344 s). One file cannot be split across shards. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, GitHub CLI read-only). Standard reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Related: #19650 (earlier repair of the weekly refresh), #21013 (Core image publish time; separate files). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
205 lines
13 KiB
JSON
205 lines
13 KiB
JSON
{
|
|
"$comment": "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.",
|
|
"unit": "ms",
|
|
"sources": [
|
|
{
|
|
"run": 43133035381,
|
|
"attempt": 1,
|
|
"sha": "42fd790e80953000a7be99eff30fb9a81bf9c7b7",
|
|
"url": "https://github.com/paperclipai/paperclip/actions/runs/43133035381"
|
|
},
|
|
{
|
|
"run": 43134251278,
|
|
"attempt": 2,
|
|
"sha": "9dfdeaad8662d9cd73a93490d09516877ef8a276",
|
|
"url": "https://github.com/paperclipai/paperclip/actions/runs/43134251278"
|
|
},
|
|
{
|
|
"run": 43145252393,
|
|
"attempt": 2,
|
|
"sha": "7d9973405c8726d6aadc7d0e6b4f6076085ebc45",
|
|
"url": "https://github.com/paperclipai/paperclip/actions/runs/43145252393"
|
|
}
|
|
],
|
|
"coverage": {
|
|
"measured": 173,
|
|
"withSetup": 173,
|
|
"inventory": 173
|
|
},
|
|
"durations": {
|
|
"server/src/__tests__/access-routes-hidden-floor.test.ts": 4459,
|
|
"server/src/__tests__/access-routes-permissions-upgrade.test.ts": 5465,
|
|
"server/src/__tests__/activity-routes.test.ts": 10563,
|
|
"server/src/__tests__/adapter-auth-signal-routes.test.ts": 9084,
|
|
"server/src/__tests__/adapter-model-refresh-routes.test.ts": 4075,
|
|
"server/src/__tests__/adapter-routes-authz.test.ts": 10176,
|
|
"server/src/__tests__/adapter-routes.test.ts": 3958,
|
|
"server/src/__tests__/agent-action-audit-routes.test.ts": 5904,
|
|
"server/src/__tests__/agent-adapter-validation-routes.test.ts": 50458,
|
|
"server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 4272,
|
|
"server/src/__tests__/agent-device-login-routes.test.ts": 6231,
|
|
"server/src/__tests__/agent-hire-auth-inheritance-routes.test.ts": 8399,
|
|
"server/src/__tests__/agent-hire-idempotency-routes.test.ts": 6320,
|
|
"server/src/__tests__/agent-instructions-routes.test.ts": 3851,
|
|
"server/src/__tests__/agent-live-run-routes.test.ts": 47196,
|
|
"server/src/__tests__/agent-permissions-routes.test.ts": 5042,
|
|
"server/src/__tests__/agent-runner-qualification-routes.test.ts": 5808,
|
|
"server/src/__tests__/agent-secrets-routes.test.ts": 5616,
|
|
"server/src/__tests__/agent-skills-routes.test.ts": 4780,
|
|
"server/src/__tests__/agent-task-hierarchy-depth-routes.test.ts": 5914,
|
|
"server/src/__tests__/agent-test-environment-routes.test.ts": 4386,
|
|
"server/src/__tests__/ai-connection-router.test.ts": 28008,
|
|
"server/src/__tests__/announcements-routes.test.ts": 4480,
|
|
"server/src/__tests__/approval-routes-idempotency.test.ts": 4100,
|
|
"server/src/__tests__/artifact-review-document-routes.test.ts": 6376,
|
|
"server/src/__tests__/assets.test.ts": 5477,
|
|
"server/src/__tests__/auth-routes.test.ts": 4182,
|
|
"server/src/__tests__/auth-session-route.test.ts": 3352,
|
|
"server/src/__tests__/authz-company-access.test.ts": 1190,
|
|
"server/src/__tests__/authz-existence-oracle-guard.test.ts": 100,
|
|
"server/src/__tests__/authz-secret-context.test.ts": 1184,
|
|
"server/src/__tests__/board-bootstrap-routes.test.ts": 5136,
|
|
"server/src/__tests__/board-chat-route-feature-flag.test.ts": 815,
|
|
"server/src/__tests__/bootstrap-claim-routes.test.ts": 3801,
|
|
"server/src/__tests__/browser-use-route-scope.test.ts": 3617,
|
|
"server/src/__tests__/built-in-agent-routes.test.ts": 9291,
|
|
"server/src/__tests__/cases-routes.test.ts": 11134,
|
|
"server/src/__tests__/cli-auth-routes.test.ts": 17169,
|
|
"server/src/__tests__/cloud-routes.test.ts": 3736,
|
|
"server/src/__tests__/companies-route-cross-company-authz.test.ts": 15423,
|
|
"server/src/__tests__/companies-route-path-guard.test.ts": 3778,
|
|
"server/src/__tests__/company-branding-route.test.ts": 6936,
|
|
"server/src/__tests__/company-default-decision-owner-route.test.ts": 6504,
|
|
"server/src/__tests__/company-import-transfer-routes.test.ts": 4904,
|
|
"server/src/__tests__/company-portability-routes.test.ts": 5250,
|
|
"server/src/__tests__/company-portability.test.ts": 9588,
|
|
"server/src/__tests__/company-search-extract-routes.test.ts": 4424,
|
|
"server/src/__tests__/company-search-rate-limit-routes.test.ts": 4412,
|
|
"server/src/__tests__/company-skill-policy-routes.test.ts": 4527,
|
|
"server/src/__tests__/company-skills-import-authz-routes.test.ts": 5740,
|
|
"server/src/__tests__/company-skills-routes.test.ts": 3796,
|
|
"server/src/__tests__/company-user-directory-route.test.ts": 3829,
|
|
"server/src/__tests__/cost-data-quality-routes.test.ts": 3649,
|
|
"server/src/__tests__/costs-service.test.ts": 5218,
|
|
"server/src/__tests__/decision-queues-routes.test.ts": 6413,
|
|
"server/src/__tests__/document-annotation-routes.test.ts": 6425,
|
|
"server/src/__tests__/email-routes-provider-errors.test.ts": 3667,
|
|
"server/src/__tests__/environment-custom-image-routes.test.ts": 3810,
|
|
"server/src/__tests__/environment-instance-routes.test.ts": 3784,
|
|
"server/src/__tests__/environment-routes.test.ts": 3862,
|
|
"server/src/__tests__/environment-selection-route-guards.test.ts": 4439,
|
|
"server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 5683,
|
|
"server/src/__tests__/execution-workspaces-routes.test.ts": 3883,
|
|
"server/src/__tests__/express5-auth-wildcard.test.ts": 120,
|
|
"server/src/__tests__/external-object-routes.test.ts": 4334,
|
|
"server/src/__tests__/folders-routes.test.ts": 3921,
|
|
"server/src/__tests__/health-dev-server-token.test.ts": 4171,
|
|
"server/src/__tests__/health.test.ts": 3773,
|
|
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 50657,
|
|
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 54768,
|
|
"server/src/__tests__/heartbeat-process-recovery.test.ts": 344017,
|
|
"server/src/__tests__/heartbeat-run-privacy-routes.test.ts": 9270,
|
|
"server/src/__tests__/inbox-agent-policy-routes.test.ts": 4707,
|
|
"server/src/__tests__/inbox-archive-routes.test.ts": 7445,
|
|
"server/src/__tests__/inbox-mine-routes.test.ts": 7381,
|
|
"server/src/__tests__/instance-database-backups-routes.test.ts": 3506,
|
|
"server/src/__tests__/instance-settings-routes.test.ts": 4187,
|
|
"server/src/__tests__/invite-accept-existing-member.test.ts": 3588,
|
|
"server/src/__tests__/invite-accept-gateway-defaults.test.ts": 4603,
|
|
"server/src/__tests__/invite-accept-replay.test.ts": 4210,
|
|
"server/src/__tests__/invite-create-route.test.ts": 6191,
|
|
"server/src/__tests__/invite-expiry.test.ts": 4468,
|
|
"server/src/__tests__/invite-join-manager.test.ts": 4095,
|
|
"server/src/__tests__/invite-list-route.test.ts": 3806,
|
|
"server/src/__tests__/invite-logo-route.test.ts": 4470,
|
|
"server/src/__tests__/invite-onboarding-text.test.ts": 4598,
|
|
"server/src/__tests__/invite-rate-limit-route.test.ts": 5758,
|
|
"server/src/__tests__/invite-summary-route.test.ts": 6219,
|
|
"server/src/__tests__/invite-test-resolution-route.test.ts": 4021,
|
|
"server/src/__tests__/invite-url-public-base-url.test.ts": 4803,
|
|
"server/src/__tests__/issue-access-grants-routes.test.ts": 7311,
|
|
"server/src/__tests__/issue-activity-events-routes.test.ts": 8154,
|
|
"server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 15844,
|
|
"server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 4282,
|
|
"server/src/__tests__/issue-assignee-invokability-routes.test.ts": 3973,
|
|
"server/src/__tests__/issue-attachment-routes.test.ts": 4512,
|
|
"server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 5372,
|
|
"server/src/__tests__/issue-chat-bootstrap-routes.test.ts": 7382,
|
|
"server/src/__tests__/issue-closed-workspace-routes.test.ts": 3977,
|
|
"server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 5540,
|
|
"server/src/__tests__/issue-comment-cancel-routes.test.ts": 3798,
|
|
"server/src/__tests__/issue-comment-reopen-routes.test.ts": 6776,
|
|
"server/src/__tests__/issue-confirm-run-stopped-routes.test.ts": 7677,
|
|
"server/src/__tests__/issue-create-deduplication-routes.test.ts": 6041,
|
|
"server/src/__tests__/issue-created-from-routes.test.ts": 5645,
|
|
"server/src/__tests__/issue-creation-source-routes.test.ts": 6585,
|
|
"server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 8660,
|
|
"server/src/__tests__/issue-document-restore-routes.test.ts": 6269,
|
|
"server/src/__tests__/issue-execution-policy-routes.test.ts": 4346,
|
|
"server/src/__tests__/issue-feedback-routes.test.ts": 4382,
|
|
"server/src/__tests__/issue-identifier-routes.test.ts": 5515,
|
|
"server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 12312,
|
|
"server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 4929,
|
|
"server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 13445,
|
|
"server/src/__tests__/issue-queued-comments-routes.test.ts": 118620,
|
|
"server/src/__tests__/issue-scheduled-retry-routes.test.ts": 14427,
|
|
"server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 7268,
|
|
"server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 6458,
|
|
"server/src/__tests__/issue-status-version-route-service.test.ts": 5591,
|
|
"server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 6142,
|
|
"server/src/__tests__/issue-summaries-routes.test.ts": 5147,
|
|
"server/src/__tests__/issue-telemetry-routes.test.ts": 4790,
|
|
"server/src/__tests__/issue-thread-interaction-routes.test.ts": 14207,
|
|
"server/src/__tests__/issue-tree-control-routes.test.ts": 3888,
|
|
"server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 4533,
|
|
"server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 5628,
|
|
"server/src/__tests__/issue-watchdogs-routes.test.ts": 11962,
|
|
"server/src/__tests__/issue-workspace-command-authz.test.ts": 4185,
|
|
"server/src/__tests__/issues-checkout-wakeup.test.ts": 100,
|
|
"server/src/__tests__/issues-goal-context-routes.test.ts": 4365,
|
|
"server/src/__tests__/issues-service.test.ts": 39512,
|
|
"server/src/__tests__/link-preview-routes.test.ts": 3783,
|
|
"server/src/__tests__/llms-routes.test.ts": 4025,
|
|
"server/src/__tests__/low-trust-red-team-routes.test.ts": 40548,
|
|
"server/src/__tests__/maintenance-server-routes.test.ts": 276429,
|
|
"server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 3791,
|
|
"server/src/__tests__/multilingual-issues-routes.test.ts": 5790,
|
|
"server/src/__tests__/muse-qualification-routes.test.ts": 3804,
|
|
"server/src/__tests__/onboarding-seed-route.test.ts": 6481,
|
|
"server/src/__tests__/openapi-routes.test.ts": 5123,
|
|
"server/src/__tests__/openclaw-invite-prompt-route.test.ts": 3811,
|
|
"server/src/__tests__/opencode-local-adapter-environment.test.ts": 6723,
|
|
"server/src/__tests__/openrouter-attribution.test.ts": 685,
|
|
"server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 6152,
|
|
"server/src/__tests__/plugin-install-route-security.test.ts": 5289,
|
|
"server/src/__tests__/plugin-routes-authz.test.ts": 3612,
|
|
"server/src/__tests__/plugin-scoped-api-routes.test.ts": 3890,
|
|
"server/src/__tests__/project-decision-owner-route.test.ts": 6221,
|
|
"server/src/__tests__/project-goal-telemetry-routes.test.ts": 4390,
|
|
"server/src/__tests__/project-routes-env.test.ts": 4228,
|
|
"server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6636,
|
|
"server/src/__tests__/projects-list-archived-routes.test.ts": 5176,
|
|
"server/src/__tests__/redaction.test.ts": 160,
|
|
"server/src/__tests__/resource-memberships-routes.test.ts": 5530,
|
|
"server/src/__tests__/routine-document-annotation-routes.test.ts": 4747,
|
|
"server/src/__tests__/routines-e2e.test.ts": 15930,
|
|
"server/src/__tests__/routines-routes.test.ts": 11937,
|
|
"server/src/__tests__/secret-proposals-routes.test.ts": 15931,
|
|
"server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 5401,
|
|
"server/src/__tests__/secrets-routes.test.ts": 4414,
|
|
"server/src/__tests__/sidebar-preferences-routes.test.ts": 6241,
|
|
"server/src/__tests__/skill-update-diff-routes.test.ts": 4160,
|
|
"server/src/__tests__/subscription-routes.test.ts": 3452,
|
|
"server/src/__tests__/summary-slot-routes.test.ts": 8150,
|
|
"server/src/__tests__/task-connector-read-routes.test.ts": 6479,
|
|
"server/src/__tests__/task-title-routes.test.ts": 13501,
|
|
"server/src/__tests__/teams-catalog-routes.test.ts": 7702,
|
|
"server/src/__tests__/user-profile-routes.test.ts": 4153,
|
|
"server/src/__tests__/workspace-command-authz.test.ts": 100,
|
|
"server/src/__tests__/workspace-preferences-routes.test.ts": 3922,
|
|
"server/src/__tests__/workspace-restore-repair-route.test.ts": 7023,
|
|
"server/src/__tests__/workspace-runtime-routes-authz.test.ts": 4185,
|
|
"server/src/__tests__/workspace-runtime-service-authz.test.ts": 2027
|
|
}
|
|
}
|