mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
refactor(perf): reuse database test setup for profile benchmark (#18683)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool profiles govern the tools that agents can use. > - Profile listing and app attention shared nested selector scans that consumed the server CPU. > - #18681 and #18673 replaced those scans with indexes for each computation. > - Their performance benchmarks need consistent setup and reproducible per-service measurements. > - This pull request uses the shared helper entry point and retains the exact measurement harness and results. > - Reviewers can inspect both the CPU savings and the limits of the evidence. ## Linked Issues or Issue Description Refs #18681 and #18673. **What existing behavior does this improve?** The consistency of benchmark setup and the reproducibility of evidence for the two merged CPU fixes. **Current behavior** The profile-review benchmark imports the database starter directly. The matching benchmark uses the shared server test-helper entry point. On the current base, that helper re-exports the database starter, which already supports `PAPERCLIP_TEST_POSTGRES_ADMIN_URL`. The previous committed benchmark measures both services together; the per-service measurements and separate HTTP/Inspector runs lacked a committed harness. **Proposed behavior** Use the same test-helper entry point in both benchmarks. Retain the compile, per-service timing, Inspector, and HTTP-probe programs with their exact commands and raw samples. Preserve the existing embedded and admin-server database behavior. **Reason and benefit** A shared entry point keeps benchmark setup consistent. The committed harness lets reviewers reproduce and compare each real owning service before and after the merged fixes. The failure-path checks verify that result-write errors still release the database and probe resources. **Breaking changes** None. This change affects benchmark setup and documentation. ## What Changed - Import the database starter from the repository test helper in `benchmark-profile-review-projection.ts`. - Record separate CPU and elapsed measurements for profile listing and app attention. - Retain the actual compile, per-service measurement, Inspector capture, and separate-process HTTP probe harness. - Always tear down the test database, HTTP server, and probe even when result writing fails. - Retain all timing and responsiveness samples as sanitized JSON. - State the runtime, fixture, source revisions, and limits of the local comparison. ## Verification - 24 focused tests passed across selector matching, pending projection, review-service freshness, and effective assignments. - Four selected real database-backed service/route tests passed for summaries, review changes, auto-allow behavior, and company isolation. The other 494 tests in that file were not selected. - Injected an `EISDIR` result-write failure into both profiling programs on an owned local admin server. Each child exited with the expected error and removed its disposable database. The responsiveness child also terminated its probe/server handles. - The changed benchmark completed with a configured local PostgreSQL admin URL. Its full responses matched the baseline, and query/row counts matched. - The exact documented compile, benchmark, and responsiveness commands completed from the committed harness at `ca458baddc6461fb6c888984246e329e2bb4bdfe`. The JSON records that build revision and the four owner source hashes. - Five alternating samples per service and implementation used Node 24.22.2 with compiled ES modules. No runtime TypeScript loader ran in those measurements. - Median process CPU from the committed harness: profile listing 3,164 ms to 385 ms; app attention 2,466 ms to 221 ms. Both versions returned identical ordered results and made seven queries per service. - Separate Inspector profiles confirmed that the repeated scope and summary scans were removed. Three separate-process HTTP probe trials per service showed less event-loop blocking. These are local synthetic service results, not hosted page-load acceptance. - Initial local setup failed on missing embedded PostgreSQL library aliases and an unbuilt plugin SDK. Restoring local aliases and building the SDK resolved those setup failures. The original failed setup logs were retained; the 28 selected tests then passed. - The merged summary fix CI run [41535304968](https://github.com/paperclipai/paperclip/actions/runs/41535304968) hit two 15-second timeouts in the unchanged attention-retention history cases. The server shard passed unchanged on one failed-jobs-only retry. Original and retry logs are retained. The compared runs used different runners and synthetic merge trees, so the original timeout cause is not established. No timeout or assertion was changed. - Full repository typecheck, test matrix, and build passed in CI on final head `2f484ed4284f3fa632d8e73d9be084827d1c987e`: 54 successful check runs, three expected skips, one non-blocking Paperclip review, and a successful Snyk status. No failures or pending checks remain. Those broad commands were not rerun locally for this benchmark-tooling follow-up; the 28 selected local regressions and the actual profiling/failure-path programs ran locally. - Fresh Greptile is 5/5 on that exact head. Both review P2 threads are resolved. The Paperclip review's current-base documentation note is corrected in this description; its optional tooling suggestions remain outside this bounded change. ## Risks - The test helper creates a unique disposable database on the configured admin server and removes it afterward. The operator must use a database server where those test operations are appropriate. - Synthetic data and shared-host activity affect absolute timings. The measurements do not establish a production latency percentage or capacity limit. - Production behavior is covered by the two linked merged fixes. This follow-up changes only benchmark setup and documentation. ## Model Used OpenAI GPT-6 through Codex, with code execution and repository tools. The exact runtime model revision and context window were not exposed in this session. The original linked performance fixes record their own model details. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
3f8ebfff8a
commit
00fea0da52
8 files changed
+875
-1
No files matched your search
@@ -0,0 +1,532 @@
|
||||
{
|
||||
"measurements": [
|
||||
{
|
||||
"kind": "environment",
|
||||
"node": "v24.22.2",
|
||||
"platform": "darwin",
|
||||
"arch": "arm64",
|
||||
"build": "esbuild transpiled ES modules, node24 target; no runtime TypeScript loader",
|
||||
"baseline": "1f8f2e195f4897cd1c237bdc068de95780ab0c3c",
|
||||
"pending": "fd59c0183ce64c44696f79604516f4d51ca5662b",
|
||||
"treatmentHead": "ca458baddc6461fb6c888984246e329e2bb4bdfe",
|
||||
"sourceSha256": {
|
||||
"profiles/projection": "fdb8b2cd220ce71a1ef322afef8d25e684beff88b73fedb1a5b73122b95e9a8d",
|
||||
"profiles/service": "d56f83ec975bf4d2955cb9a31e9b4ebeb4f3872f28d930436bd37b6004ce1824",
|
||||
"profiles/resolution": "cb621da4864ee143922c94644487b8bf53a185848a419e9b05945ed6b0237a14",
|
||||
"catalog/attention": "c3fdc44e01f39b59cec047584c75d99df192f4e67809a6078ac75dfbb3c2b005"
|
||||
},
|
||||
"forcedGc": true,
|
||||
"fixture": {
|
||||
"profiles": 200,
|
||||
"activeTools": 1900,
|
||||
"removedTools": 695,
|
||||
"rules": 76000,
|
||||
"agents": 12
|
||||
},
|
||||
"repeats": 5,
|
||||
"caveat": "Local synthetic service workload, shared developer host; not hosted page-load acceptance"
|
||||
},
|
||||
{
|
||||
"kind": "correctness",
|
||||
"owner": "profiles",
|
||||
"completeOutputEquality": true
|
||||
},
|
||||
{
|
||||
"kind": "correctness",
|
||||
"owner": "attention",
|
||||
"completeOutputEquality": true
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 0,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"wallMs": 3044.6725000000006,
|
||||
"cpuMs": 3086.254,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 0,
|
||||
"owner": "profiles",
|
||||
"label": "pending",
|
||||
"wallMs": 1331.632625000002,
|
||||
"cpuMs": 1475.759,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 0,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"wallMs": 214.5939580000013,
|
||||
"cpuMs": 384.783,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 0,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"wallMs": 2378.09475,
|
||||
"cpuMs": 2466.076,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 0,
|
||||
"owner": "attention",
|
||||
"label": "pending",
|
||||
"wallMs": 174.1545839999999,
|
||||
"cpuMs": 319.323,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 0,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"wallMs": 159.26837499999965,
|
||||
"cpuMs": 219.179,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 1,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"wallMs": 181.5132080000003,
|
||||
"cpuMs": 224.232,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 1,
|
||||
"owner": "profiles",
|
||||
"label": "pending",
|
||||
"wallMs": 1997.141208000001,
|
||||
"cpuMs": 1830.433,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 1,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"wallMs": 4942.23775,
|
||||
"cpuMs": 4553.281,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 1,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"wallMs": 318.5823750000018,
|
||||
"cpuMs": 466.636,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 1,
|
||||
"owner": "attention",
|
||||
"label": "pending",
|
||||
"wallMs": 192.32737500000076,
|
||||
"cpuMs": 254.286,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 1,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"wallMs": 2694.246791999998,
|
||||
"cpuMs": 2761.568,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 2,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"wallMs": 4083.2392089999994,
|
||||
"cpuMs": 3864.952,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 2,
|
||||
"owner": "profiles",
|
||||
"label": "pending",
|
||||
"wallMs": 1423.3007910000015,
|
||||
"cpuMs": 1581.88,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 2,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"wallMs": 223.0350000000035,
|
||||
"cpuMs": 414.014,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 2,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"wallMs": 1864.8400000000038,
|
||||
"cpuMs": 1928.275,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 2,
|
||||
"owner": "attention",
|
||||
"label": "pending",
|
||||
"wallMs": 175.5971669999999,
|
||||
"cpuMs": 339.946,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 2,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"wallMs": 147.17741699999897,
|
||||
"cpuMs": 202.889,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 3,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"wallMs": 182.54275000000052,
|
||||
"cpuMs": 218.628,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 3,
|
||||
"owner": "profiles",
|
||||
"label": "pending",
|
||||
"wallMs": 1203.7079159999994,
|
||||
"cpuMs": 1258.248,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 3,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"wallMs": 2998.833458000001,
|
||||
"cpuMs": 3164.455,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 3,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"wallMs": 174.7018339999995,
|
||||
"cpuMs": 353.529,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 3,
|
||||
"owner": "attention",
|
||||
"label": "pending",
|
||||
"wallMs": 147.98641700000007,
|
||||
"cpuMs": 204.736,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 3,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"wallMs": 2482.0454580000005,
|
||||
"cpuMs": 2538.083,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 4,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"wallMs": 2980.427125000002,
|
||||
"cpuMs": 3091.659,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 4,
|
||||
"owner": "profiles",
|
||||
"label": "pending",
|
||||
"wallMs": 1334.6162909999985,
|
||||
"cpuMs": 1494.868,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 4,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"wallMs": 233.16475000000355,
|
||||
"cpuMs": 439.851,
|
||||
"queries": 7,
|
||||
"rows": 78114,
|
||||
"resultCount": 200
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 4,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"wallMs": 2031.9052500000034,
|
||||
"cpuMs": 2090.426,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 4,
|
||||
"owner": "attention",
|
||||
"label": "pending",
|
||||
"wallMs": 173.21041600000171,
|
||||
"cpuMs": 316.039,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "timing",
|
||||
"round": 4,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"wallMs": 158.0989999999947,
|
||||
"cpuMs": 221.091,
|
||||
"queries": 7,
|
||||
"rows": 78101
|
||||
},
|
||||
{
|
||||
"kind": "profile",
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"durationMs": 2994.209,
|
||||
"samples": 2345
|
||||
},
|
||||
{
|
||||
"kind": "profile",
|
||||
"owner": "profiles",
|
||||
"label": "pending",
|
||||
"durationMs": 1304.667,
|
||||
"samples": 999
|
||||
},
|
||||
{
|
||||
"kind": "profile",
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"durationMs": 271.542,
|
||||
"samples": 173
|
||||
},
|
||||
{
|
||||
"kind": "profile",
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"durationMs": 2502.958,
|
||||
"samples": 1958
|
||||
},
|
||||
{
|
||||
"kind": "profile",
|
||||
"owner": "attention",
|
||||
"label": "pending",
|
||||
"durationMs": 248.75,
|
||||
"samples": 149
|
||||
},
|
||||
{
|
||||
"kind": "profile",
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"durationMs": 216.375,
|
||||
"samples": 129
|
||||
}
|
||||
],
|
||||
"responsiveness": [
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 0,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"operationMs": 3020.712625,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 2885.9768750000003,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 0,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"operationMs": 216.8745829999989,
|
||||
"completedHealthRequests": 5,
|
||||
"maxHealthMs": 78.3608750000003,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 0,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"operationMs": 2220.037542,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 2103.2218329999996,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 0,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"operationMs": 178.0747500000016,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 55.81975000000057,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 1,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"operationMs": 180.5625,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 63.572291000000405,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 1,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"operationMs": 3155.7248339999987,
|
||||
"completedHealthRequests": 5,
|
||||
"maxHealthMs": 3027.892249999999,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 1,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"operationMs": 183.60120800000004,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 71.33616699999766,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 1,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"operationMs": 2413.9667499999996,
|
||||
"completedHealthRequests": 3,
|
||||
"maxHealthMs": 2292.3195000000014,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 2,
|
||||
"owner": "profiles",
|
||||
"label": "baseline",
|
||||
"operationMs": 3146.798458000001,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 3008.1585419999974,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 2,
|
||||
"owner": "profiles",
|
||||
"label": "combined",
|
||||
"operationMs": 230.38120900000285,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 83.7714589999996,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 2,
|
||||
"owner": "attention",
|
||||
"label": "baseline",
|
||||
"operationMs": 1886.0740839999999,
|
||||
"completedHealthRequests": 4,
|
||||
"maxHealthMs": 1770.5739170000015,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
},
|
||||
{
|
||||
"kind": "responsiveness",
|
||||
"round": 2,
|
||||
"owner": "attention",
|
||||
"label": "combined",
|
||||
"operationMs": 180.26520900000105,
|
||||
"completedHealthRequests": 3,
|
||||
"maxHealthMs": 64.22629199999938,
|
||||
"errors": [],
|
||||
"scope": "Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route"
|
||||
}
|
||||
],
|
||||
"responsiveness_limit": "Minimal local HTTP liveness handler on the same event loop as real database-backed service operations, not the assembled application health route."
|
||||
}
|
||||
@@ -138,3 +138,86 @@ seven samples of ten resolutions. It reports actual query and returned-row
|
||||
counts, process CPU, and elapsed time. The installation dependency returns an
|
||||
empty list for this fixture, which has no installed connections. It does not
|
||||
measure provider work, whole-run latency, or hosted performance.
|
||||
|
||||
|
||||
## Local profile comparison on 2026-11-02
|
||||
|
||||
The profile and app-attention paths were re-profiled together after
|
||||
[#18681](https://github.com/paperclipai/paperclip/pull/18681) and
|
||||
[#18673](https://github.com/paperclipai/paperclip/pull/18673). Both fixes replace
|
||||
nested selector scans with indexes that live for one computation.
|
||||
|
||||
The comparison used Node 24.22.2 on macOS arm64. Service modules were compiled
|
||||
as ES modules for Node 24 before measurement. No TypeScript loader ran during
|
||||
measurement. All variants used the same database, fixture, and database client.
|
||||
Only the service and projection implementation changed between variants.
|
||||
The fixture had 200 profiles, 1,900 active tools, 695 removed tools, 76,000
|
||||
rules, and 12 agents. Complete ordered responses matched before timing.
|
||||
|
||||
Baseline: `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. The pending-review fix was
|
||||
`fd59c0183ce64c44696f79604516f4d51ca5662b`. The summary fix was
|
||||
`e20165b6eb871c29e504a3837e8fe92a17ef969f`; its later benchmark helper change
|
||||
does not change production code. Five unprofiled samples alternated order,
|
||||
with forced GC before each operation. Values below are medians.
|
||||
[All samples](benchmarks/2026-11-02-profile-projection.json) retain each measurement
|
||||
and the pending-review-only treatment.
|
||||
|
||||
| Service | Baseline CPU | Combined CPU | Baseline elapsed | Combined elapsed |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| `listProfiles` | 3,164 ms | 385 ms | 3,045 ms | 215 ms |
|
||||
| `listAppsNeedingAttention` | 2,466 ms | 221 ms | 2,378 ms | 159 ms |
|
||||
|
||||
Each version made seven queries per service. Profile listing returned 78,114
|
||||
database rows; app attention returned 78,101. These counts did not change.
|
||||
The gains came from less JavaScript work after the reads.
|
||||
|
||||
Separate Inspector CPU captures used a 1 ms sampling interval. The baseline
|
||||
showed `profileCoversCatalogScope` and `summarizeProfile` as major CPU costs.
|
||||
Their repeated scans were absent from the combined capture. Residual work
|
||||
included pending projection, row conversion, and garbage collection.
|
||||
Profile capture time was excluded from the timing samples above.
|
||||
|
||||
The committed harness is in `server/scripts/profile-tool-projection/`.
|
||||
Use Node 24.22.2, install workspace dependencies, and run these commands from
|
||||
the repository root in a disposable worktree. The compile step writes local
|
||||
`dist/` output. It is not the full application build. The Git history must
|
||||
contain the baseline and pending-fix commits named above.
|
||||
|
||||
```sh
|
||||
git fetch https://github.com/paperclipai/paperclip.git refs/pull/18681/head
|
||||
pnpm install --frozen-lockfile
|
||||
node server/scripts/profile-tool-projection/compile.mjs
|
||||
node --expose-gc --import ./server/scripts/profile-tool-projection/register.mjs server/scripts/profile-tool-projection/benchmark.mjs --output-dir /tmp/paperclip-tool-projection --repeat 5
|
||||
node --expose-gc --import ./server/scripts/profile-tool-projection/register.mjs server/scripts/profile-tool-projection/responsiveness.mjs --output-dir /tmp/paperclip-tool-projection
|
||||
```
|
||||
|
||||
The compile step writes the baseline and pending-only owner modules beside
|
||||
the current compiled modules. A build manifest records the current Git HEAD
|
||||
and the SHA-256 hashes of the four current owner source files. The benchmark
|
||||
records this manifest, validates complete output equality, then measures
|
||||
`profiles` and `attention` separately for `baseline`, `pending`, and `combined`.
|
||||
It writes `measurements.json` and six separate `.cpuprofile` files to the
|
||||
output directory. The HTTP probe writes `responsiveness.json` there. Keep
|
||||
CPU profiles private because they contain local source paths. Rebuild and
|
||||
rerun if the owner files change. Serialize these commands with other heavy
|
||||
builds and benchmarks on the same host.
|
||||
|
||||
A separate process also sent simple HTTP requests every 20 ms while each
|
||||
real service operation ran. The local liveness handler shared the service's
|
||||
Node event loop. There was at most one outstanding probe request. Across
|
||||
three alternating trials, the median largest probe delay fell from 3,008 ms
|
||||
to 78 ms during profile listing, and from 2,103 ms to 64 ms during app
|
||||
attention. All 48 probe requests completed without error.
|
||||
|
||||
These are synthetic service measurements on a shared developer machine.
|
||||
They are not a full application build, a browser test, or hosted acceptance.
|
||||
The liveness handler is a minimal test handler, not the application's health
|
||||
route. Workload, JIT warmup, and host activity affect absolute timings. The
|
||||
comparison does not establish a page-load percentage or production capacity.
|
||||
Use a matched deployment window to confirm the effect on user requests.
|
||||
|
||||
Both benchmark scripts use the test database helper. Set
|
||||
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to use a local PostgreSQL server; otherwise
|
||||
they create and remove an embedded test database. The helper creates a unique
|
||||
disposable database and removes it afterward. Benchmark setup and fixture
|
||||
insertion are outside the timing samples.
|
||||
@@ -1,7 +1,8 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { createDb, startEmbeddedPostgresTestDatabase, type Db } from "@paperclipai/db";
|
||||
import { createDb, type Db } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../src/__tests__/helpers/embedded-postgres.js";
|
||||
import { createToolProfileService } from "../src/services/tool-access/profiles/service.js";
|
||||
import { createToolAppAttention } from "../src/services/tool-access/catalog/attention.js";
|
||||
import { createProfileProjectionFixture } from "../src/__tests__/helpers/tool-profile-projection-fixture.js";
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { writeFile, mkdir, readFile } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { Session } from 'node:inspector';
|
||||
import { createDb } from '@paperclipai/db';
|
||||
import { startEmbeddedPostgresTestDatabase } from '../../dist/__tests__/helpers/embedded-postgres.js';
|
||||
import { createProfileProjectionFixture } from '../../dist/__tests__/helpers/tool-profile-projection-fixture.js';
|
||||
const option = (name, fallback) => { const index = process.argv.indexOf(name); return index < 0 ? fallback : process.argv[index + 1] ?? fallback; };
|
||||
const output = path.resolve(option('--output-dir', '/tmp/paperclip-tool-projection'));
|
||||
const repeats = Number(option('--repeat', '5'));
|
||||
assert.ok(Number.isInteger(repeats) && repeats > 0, '--repeat must be a positive integer');
|
||||
assert.equal(typeof global.gc, 'function', 'Run Node with --expose-gc');
|
||||
const build = JSON.parse(await readFile(new URL('../../dist/.profile-projection-build.json', import.meta.url), 'utf8'));
|
||||
await mkdir(output, { recursive: true });
|
||||
const variants = {};
|
||||
for (const label of ['baseline', 'pending', 'combined']) {
|
||||
const p = label === 'combined' ? './service.js' : `./.${label}-service.js`;
|
||||
const a = label === 'combined' ? './attention.js' : `./.${label}-attention.js`;
|
||||
variants[label] = {
|
||||
profileFactory: (await import(new URL(`../../dist/services/tool-access/profiles/${p}`, import.meta.url))).createToolProfileService,
|
||||
attentionFactory: (await import(new URL(`../../dist/services/tool-access/catalog/${a}`, import.meta.url))).createToolAppAttention,
|
||||
};
|
||||
}
|
||||
const database = await startEmbeddedPostgresTestDatabase('paperclip-hotpaths-owned-');
|
||||
const db = createDb(database.connectionString, { maxConnections: 1 });
|
||||
let queries = 0, rows = 0;
|
||||
const measuredDb = new Proxy(db, { get(target, key) {
|
||||
if (key !== 'select') return Reflect.get(target, key);
|
||||
return (...fields) => {
|
||||
const builder = target.select(...fields), from = builder.from.bind(builder);
|
||||
builder.from = (...args) => {
|
||||
const selection = from(...args), execute = selection.execute.bind(selection);
|
||||
selection.execute = async () => { const result = await execute(); queries++; rows += result.length; return result; };
|
||||
return selection;
|
||||
};
|
||||
return builder;
|
||||
};
|
||||
} });
|
||||
const records = [];
|
||||
const emit = record => { records.push(record); console.log(JSON.stringify(record)); };
|
||||
try {
|
||||
const fixture = await createProfileProjectionFixture(db);
|
||||
const now = () => new Date('2026-02-01T00:00:00Z');
|
||||
for (const variant of Object.values(variants)) {
|
||||
const profiles = variant.profileFactory(measuredDb, now);
|
||||
const attention = variant.attentionFactory(measuredDb, now);
|
||||
variant.profiles = () => profiles.listProfiles(fixture.company.id);
|
||||
variant.attention = () => attention(fixture.company.id);
|
||||
}
|
||||
emit({ kind: 'environment', node: process.version, platform: process.platform, arch: process.arch,
|
||||
build: 'esbuild transpiled ES modules, node24 target; no runtime TypeScript loader',
|
||||
...build, forcedGc: true,
|
||||
fixture: { profiles: 200, activeTools: 1900, removedTools: 695, rules: 76000, agents: 12 }, repeats,
|
||||
caveat: 'Local synthetic service workload, shared developer host; not hosted page-load acceptance' });
|
||||
for (const owner of ['profiles', 'attention']) {
|
||||
const expected = await variants.baseline[owner]();
|
||||
for (const label of ['pending', 'combined']) assert.deepEqual(await variants[label][owner](), expected);
|
||||
emit({ kind: 'correctness', owner, completeOutputEquality: true });
|
||||
}
|
||||
for (let round = 0; round < repeats; round++) {
|
||||
for (const owner of ['profiles', 'attention']) {
|
||||
const order = round % 2 ? ['combined', 'pending', 'baseline'] : ['baseline', 'pending', 'combined'];
|
||||
for (const label of order) {
|
||||
global.gc?.(); queries = 0; rows = 0;
|
||||
const started = performance.now(), cpu = process.cpuUsage();
|
||||
const result = await variants[label][owner]();
|
||||
const used = process.cpuUsage(cpu);
|
||||
emit({ kind: 'timing', round, owner, label, wallMs: performance.now() - started, cpuMs: (used.user + used.system) / 1000,
|
||||
queries, rows, resultCount: owner === 'profiles' ? result.length : result.items?.length });
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const owner of ['profiles', 'attention']) {
|
||||
for (const label of ['baseline', 'pending', 'combined']) {
|
||||
global.gc?.();
|
||||
const session = new Session(); session.connect();
|
||||
const post = (method, args = {}) => new Promise((resolve, reject) => session.post(method, args, (error, value) => error ? reject(error) : resolve(value)));
|
||||
try {
|
||||
await post('Profiler.enable'); await post('Profiler.setSamplingInterval', { interval: 1000 });
|
||||
await post('Profiler.start');
|
||||
await variants[label][owner]();
|
||||
const { profile } = await post('Profiler.stop');
|
||||
await writeFile(`${output}/${owner}-${label}.cpuprofile`, JSON.stringify(profile));
|
||||
emit({ kind: 'profile', owner, label, durationMs: (profile.endTime - profile.startTime) / 1000, samples: profile.samples?.length });
|
||||
} finally { session.disconnect(); }
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
try {
|
||||
await writeFile(`${output}/measurements.json`, JSON.stringify(records, null, 2));
|
||||
} finally {
|
||||
try { await db.$client.end({ timeout: 1 }); }
|
||||
finally { await database.cleanup(); }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
import { build, transform } from 'esbuild';
|
||||
import { readdir, mkdir, writeFile, cp, readFile } from 'node:fs/promises';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import path from 'node:path';
|
||||
const root = fileURLToPath(new URL("../../..", import.meta.url));
|
||||
process.chdir(root);
|
||||
async function sources(dir) {
|
||||
const out = [];
|
||||
for (const entry of await readdir(dir, { withFileTypes: true })) {
|
||||
const file = path.join(dir, entry.name);
|
||||
if (entry.isDirectory() && entry.name !== '__tests__') out.push(...await sources(file));
|
||||
else if (entry.isFile() && /\.tsx?$/.test(entry.name) && !/\.d\.ts$|\.test\.tsx?$/.test(entry.name)) out.push(file);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
async function assets(dir, out) {
|
||||
for (const entry of await readdir(dir, { withFileTypes: true })) {
|
||||
const file = path.join(dir, entry.name), target = path.join(out, entry.name);
|
||||
if (entry.isDirectory() && entry.name !== '__tests__') await assets(file, target);
|
||||
else if (entry.isFile() && entry.name.endsWith('.json')) { await mkdir(out, { recursive: true }); await cp(file, target); }
|
||||
}
|
||||
}
|
||||
for (const pkg of ['packages/shared', 'packages/db', 'server']) {
|
||||
const src = path.join(root, pkg, 'src');
|
||||
await build({ entryPoints: await sources(src), outbase: src, outdir: path.join(root, pkg, 'dist'),
|
||||
format: 'esm', platform: 'node', target: 'node24', sourcemap: true, logLevel: 'warning' });
|
||||
await assets(src, path.join(root, pkg, 'dist'));
|
||||
}
|
||||
await cp('packages/db/src/migrations', 'packages/db/dist/migrations', { recursive: true });
|
||||
await cp('packages/shared/src/app-definitions', 'packages/shared/dist/app-definitions', { recursive: true });
|
||||
await mkdir('server/dist/__tests__/helpers', { recursive: true });
|
||||
await build({ entryPoints: ['server/src/__tests__/helpers/tool-profile-projection-fixture.ts', 'server/src/__tests__/helpers/embedded-postgres.ts'],
|
||||
outbase: 'server/src/__tests__/helpers', outdir: 'server/dist/__tests__/helpers', format: 'esm', platform: 'node', target: 'node24', sourcemap: true });
|
||||
for (const [label, sha] of [['baseline', '1f8f2e195f4897cd1c237bdc068de95780ab0c3c'], ['pending', 'fd59c0183ce64c44696f79604516f4d51ca5662b']]) {
|
||||
for (const [name, file] of [['projection', 'profiles/projection'], ['service', 'profiles/service'], ['attention', 'catalog/attention']]) {
|
||||
const relative = `server/src/services/tool-access/${file}.ts`;
|
||||
let code = execFileSync('git', ['show', `${sha}:${relative}`], { encoding: 'utf8', maxBuffer: 5e6 });
|
||||
if (name === 'service') code = code.replace('from "./projection.js"', `from "./.${label}-projection.js"`);
|
||||
if (name === 'attention') code = code.replace('from "../profiles/projection.js"', `from "../profiles/.${label}-projection.js"`);
|
||||
const outfile = `server/dist/services/tool-access/${path.dirname(file)}/.${label}-${name}.js`;
|
||||
const result = await transform(code, { loader: 'ts', format: 'esm', platform: 'node', target: 'node24', sourcefile: relative, sourcemap: 'inline' });
|
||||
await writeFile(outfile, result.code);
|
||||
}
|
||||
}
|
||||
const owners = ['profiles/projection', 'profiles/service', 'profiles/resolution', 'catalog/attention'];
|
||||
const manifest = {
|
||||
baseline: '1f8f2e195f4897cd1c237bdc068de95780ab0c3c',
|
||||
pending: 'fd59c0183ce64c44696f79604516f4d51ca5662b',
|
||||
treatmentHead: execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(),
|
||||
sourceSha256: Object.fromEntries(await Promise.all(owners.map(async file => [file,
|
||||
createHash('sha256').update(await readFile(`server/src/services/tool-access/${file}.ts`)).digest('hex')]))),
|
||||
};
|
||||
await writeFile('server/dist/.profile-projection-build.json', JSON.stringify(manifest, null, 2));
|
||||
console.log('Compiled baseline, pending-only, and current owning service modules; no runtime TypeScript loader.');
|
||||
@@ -0,0 +1,20 @@
|
||||
import http from 'node:http';
|
||||
const agent = new http.Agent({ keepAlive: true });
|
||||
let active = false, pending, times = [], port;
|
||||
function ping() {
|
||||
if (!active || pending) return;
|
||||
pending = new Promise(resolve => {
|
||||
const start = performance.now();
|
||||
const request = http.get({ host: '127.0.0.1', port, path: '/api/health', agent }, response => {
|
||||
response.resume(); response.on('end', () => { times.push(performance.now() - start); resolve(); });
|
||||
});
|
||||
request.setTimeout(20000, () => request.destroy(new Error('Probe timeout')));
|
||||
request.on('error', error => { times.push({ error: error.message }); resolve(); });
|
||||
}).finally(() => { pending = null; });
|
||||
}
|
||||
const interval = setInterval(ping, 20);
|
||||
process.on('message', async message => {
|
||||
if (message.kind === 'arm') { port = message.port; times = []; active = true; process.send({ kind: 'armed' }); }
|
||||
if (message.kind === 'stop') { active = false; await pending; process.send({ kind: 'stopped', times }); }
|
||||
if (message.kind === 'close') { clearInterval(interval); agent.destroy(); process.disconnect(); }
|
||||
});
|
||||
@@ -0,0 +1,13 @@
|
||||
import { registerHooks } from 'node:module';
|
||||
import { pathToFileURL, fileURLToPath } from 'node:url';
|
||||
import path from 'node:path';
|
||||
const root = fileURLToPath(new URL("../../..", import.meta.url));
|
||||
registerHooks({ resolve(specifier, context, next) {
|
||||
for (const [name, dir] of [['@paperclipai/db', 'packages/db'], ['@paperclipai/shared', 'packages/shared']]) {
|
||||
if (specifier === name || specifier.startsWith(name + '/')) {
|
||||
const file = specifier === name ? 'index' : specifier.slice(name.length + 1);
|
||||
return next(pathToFileURL(path.join(root, dir, 'dist', `${file}.js`)).href, context);
|
||||
}
|
||||
}
|
||||
return next(specifier, context);
|
||||
} });
|
||||
@@ -0,0 +1,74 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import path from 'node:path';
|
||||
import { createServer } from 'node:http';
|
||||
import { fork } from 'node:child_process';
|
||||
import { once } from 'node:events';
|
||||
import { writeFile, mkdir } from 'node:fs/promises';
|
||||
import { createDb } from '@paperclipai/db';
|
||||
import { startEmbeddedPostgresTestDatabase } from '../../dist/__tests__/helpers/embedded-postgres.js';
|
||||
import { createProfileProjectionFixture } from '../../dist/__tests__/helpers/tool-profile-projection-fixture.js';
|
||||
const index = process.argv.indexOf('--output-dir');
|
||||
const output = path.resolve(index < 0 ? '/tmp/paperclip-tool-projection' : process.argv[index + 1]);
|
||||
assert.equal(typeof global.gc, 'function', 'Run Node with --expose-gc');
|
||||
await mkdir(output, { recursive: true });
|
||||
const database = await startEmbeddedPostgresTestDatabase('paperclip-hotpaths-http-owned-');
|
||||
const db = createDb(database.connectionString, { maxConnections: 1 });
|
||||
const server = createServer((_request, response) => { response.writeHead(200, { 'Content-Type': 'application/json' }); response.end('{"status":"ok"}'); });
|
||||
let probe;
|
||||
const control = async message => {
|
||||
const reply = once(probe, 'message', { signal: AbortSignal.timeout(30000) });
|
||||
probe.send(message);
|
||||
return (await reply)[0];
|
||||
};
|
||||
const records = [];
|
||||
try {
|
||||
server.listen(0, '127.0.0.1'); await once(server, 'listening');
|
||||
probe = fork(new URL('./probe.mjs', import.meta.url), { execArgv: [], stdio: ['ignore', 'inherit', 'inherit', 'ipc'] });
|
||||
const fixture = await createProfileProjectionFixture(db);
|
||||
const variants = {}, now = () => new Date('2026-02-01T00:00:00Z');
|
||||
for (const label of ['baseline', 'combined']) {
|
||||
const p = label === 'combined' ? 'service' : '.baseline-service';
|
||||
const a = label === 'combined' ? 'attention' : '.baseline-attention';
|
||||
const profiles = (await import(new URL(`../../dist/services/tool-access/profiles/${p}.js`, import.meta.url))).createToolProfileService(db, now);
|
||||
const attention = (await import(new URL(`../../dist/services/tool-access/catalog/${a}.js`, import.meta.url))).createToolAppAttention(db, now);
|
||||
variants[label] = { profiles: () => profiles.listProfiles(fixture.company.id), attention: () => attention(fixture.company.id) };
|
||||
}
|
||||
for (const variant of Object.values(variants)) for (const owner of ['profiles', 'attention']) await variant[owner]();
|
||||
for (let round = 0; round < 3; round++) {
|
||||
for (const owner of ['profiles', 'attention']) {
|
||||
for (const label of round % 2 ? ['combined', 'baseline'] : ['baseline', 'combined']) {
|
||||
global.gc?.();
|
||||
await control({ kind: 'arm', port: server.address().port });
|
||||
const start = performance.now();
|
||||
await variants[label][owner]();
|
||||
const operationMs = performance.now() - start;
|
||||
const result = await control({ kind: 'stop' });
|
||||
const good = result.times.filter(value => typeof value === 'number');
|
||||
const record = { kind: 'responsiveness', round, owner, label, operationMs,
|
||||
completedHealthRequests: good.length, maxHealthMs: good.length ? Math.max(...good) : null,
|
||||
errors: result.times.filter(value => typeof value !== 'number'),
|
||||
scope: 'Separate-process HTTP probe to a minimal liveness handler in the same Node event loop as the real service; not the full application health route' };
|
||||
records.push(record); console.log(JSON.stringify(record));
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
try {
|
||||
await writeFile(path.join(output, 'responsiveness.json'), JSON.stringify(records, null, 2));
|
||||
} finally {
|
||||
try {
|
||||
if (probe && probe.exitCode === null && probe.signalCode === null) {
|
||||
const exited = once(probe, 'exit');
|
||||
if (probe.connected) probe.send({ kind: 'close' });
|
||||
else probe.kill();
|
||||
await exited;
|
||||
}
|
||||
} finally {
|
||||
try { await new Promise(resolve => server.close(resolve)); }
|
||||
finally {
|
||||
try { await db.$client.end({ timeout: 1 }); }
|
||||
finally { await database.cleanup(); }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user