Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 b5ba548112 feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line
that a real-agent ramp harness checks (#21695)
> - Four finish-line criteria need numbers from inside each server
process: main-thread CPU, run event lateness, agent API time before the
handler, and missed lease renewals
> - The server had no way to give them, so every harness run was at best
"incomplete"
> - This pull request adds one instance-admin endpoint that returns
those numbers per process over a short sliding window, off by default,
and connects the harness to it
> - The benefit is that a ramp run can measure every server criterion it
can see, with near-zero cost when on and no cost when off

## Linked Issues or Issue Description

Refs #21695

**Subsystem affected**

Server (instance routes, run event append, native lease renewal),
database client (statement counter), and the scale-ramp harness in
`scripts/scale-ramp/`.

**Problem or motivation**

The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the
server has no endpoint for it. The criteria `main_thread_cpu`,
`run_event_lateness_p99`, `agent_api_pre_handler_p99` and
`lease_renewals_missed_zero` are therefore always "not measured", and no
run can pass.

**Proposed solution**

Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of
one server process over the last 50 to 60 s. Only an instance admin can
read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start,
default off) turns on collection and the route. Off, the path answers
404 and no collector starts.

**Alternatives considered**

- The postgres.js `debug` option for the statement counter: rejected,
because it captures a stack (`new Error()`) for every query.
- Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected,
because it includes GC and libuv threads. It is returned as `processCpu`
instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux
only, null elsewhere).
- Default on: the measured cost is negligible (numbers below), but the
event-loop monitor wakes the process every 20 ms when idle, and this is
a new instance-wide surface. It stays off; the runbook says how to turn
it on for a ramp run.

## What Changed

- `server/src/services/server-metrics.ts`: a per-process collector. It
uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20
ms), `process.cpuUsage()`, the main thread's
`/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native
`perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing
on the hot path awaits, allocates per call or does I/O.
- `server/src/routes/instance-server-metrics.ts`: `GET
/api/instance/server-metrics`, guarded by the existing
`assertInstanceAdmin` (the check that `/instance/database-backups` and
other instance routes use). It returns numbers, a random per-process
`processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no
SQL, names, IDs or hostnames.
- `server/src/app.ts`: when the flag is on, two middlewares time agent
API requests from arrival to the API router, the route is mounted, and
shutdown stops the collector. When the flag is off, nothing is mounted.
- The snapshot also carries `dedicatedConnections` from
`dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at
request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit`
since start. The harness ignores it today; it shows dedicated-connection
queueing in a restart burst.
- `packages/db/src/statement-count.ts` and
`lazy-transaction-queries.ts`: a Drizzle logger that increments one
counter per statement replaces Drizzle's no-op logger. No edit to
`client.ts`.
- `paperclip-control-plane-port.ts`: one call records runner `emittedAt`
to committed row for each committed (not duplicate) runner event.
- `native-execution-lease.ts`: one call counts a lease whose renewal
deadline passed.
- Harness: missed lease renewals are summed per `processId` (several
processes, or a restart, no longer cancel out). A metrics URL on the
target origin gets the board token in the `Authorization` header;
`PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin
gets no token.
- Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the
serializer's own output. A server test asserts the serializer produces
exactly this file, and a harness test parses it and checks every field.
- `rules/company-scope-exceptions.mdc`: adds the snapshot as an
instance-wide exception with its reason. The runbook in the plan doc
documents the path, the flag, the credential and each field.

| Field | Provided | Source |
| --- | --- | --- |
| `mainThreadCpu` | yes (Linux; null elsewhere) |
`/proc/self/task/<pid>/stat` utime+stime |
| `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` |
| `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s
p99 in the window |
| `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT
or raw driver calls) |
| `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue
time |
| `runEventLatenessP99Ms` | yes (null with no events) | runner event
append |
| `agentApiPreHandlerP99Ms` | yes (null with no agent requests) |
arrival and router-entry middlewares |
| `leaseRenewalsMissed` | yes, counter since process start | native
lease deadline timer |

## Verification

- `cd server && npx vitest run
src/__tests__/instance-server-metrics-routes.test.ts
src/__tests__/server-metrics.test.ts`: through `createApp` on the
embedded test database. 200 with the exact field set for an instance
admin; 403 for a company owner who is not an instance admin, for an
agent key, and unauthenticated; 404 and no collector when the flag is
off; values move after agent requests, statements, a late run event and
a missed renewal; the body has no IDs, keys or SQL.
- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks
(red/green checked for the lease hook by removing the call).
- `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts
src/client-options.test.ts`: the counter counts statements inside
transactions and savepoints.
- `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p
scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture
contract and the board-token header.
- `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in
`packages/db`: pass.

Overhead, measured on a shared, loaded laptop (load average 8 to 20),
median with min and max:

| Measure | Off | On |
| --- | --- | --- |
| Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) |
3.36 ns (3.32 to 6.16) |
| CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us
(20.3 to 42.4) | 32.3 us (20.7 to 45.5) |
| CPU per HTTP request, client and server in one process (2,000 x 6) |
67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) |
| Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08%
(0.05 to 0.09) |
| Record one run event (Date.parse + histogram) | - | 148 ns (135 to
188) |
| One snapshot | - | 55 us |

The statement and request differences are inside the run-to-run spread.
The direct cost is about 3 ns per statement, against about 30 us of CPU
per statement.

## Risks

- Low risk. Off by default; off, only the statement counter runs, and it
costs one increment more than Drizzle's no-op logger.
- `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other
platforms. The harness then uses `eventLoopUtilization`.
- Run event lateness compares the runner's clock with the server's
clock, so clock skew adds to it. Negative values count as zero.
- With several processes, each request reaches one process. The harness
keeps the highest gauge and sums counter growth per process, but a
process it never samples is not seen.
- The statement counter does not count BEGIN/COMMIT or raw postgres.js
calls.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:13:45 -06:00
DottaandClaude b5f62b3384 docs(rules): correct the scope of the API route and company-scope rules (#20596)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `rules/*.mdc` holds the architecture rules that reviewers and agents
apply to a change
> - Pull request #20585 added two rules: API route expectations and
company-scope exceptions
> - Review found that both rules state more than is true for the current
code
> - A rule that is too broad makes reviewers flag correct code and can
make an agent add a wrong check
> - This pull request corrects the scope of both rules and fixes one
stale pointer
> - The benefit is rules that match the code they describe

## Linked Issues or Issue Description

Refs #20585

**Issue type**

Incorrect contributor documentation.

**Where is the issue?**

`rules/api-route-expectations.mdc`,
`rules/company-scope-exceptions.mdc`, `AGENTS.md` section 8, and
`.agents/skills/honeycomb-tracing/SKILL.md`.

**What's wrong?**

- The API route rule said that every route checks company access.
Instance-level routes such as `health.ts`, `liveness.ts`, `ready.ts`,
`public-mcp.ts` and `org-chart-svg.ts` do not, and must not.
- The API route rule told reviewers to flag each status code outside a
list of seven. `429` and `503` are correct in some routes.
- The company-scope rule said that it lists the only instance-wide data.
`user_sidebar_preferences` is instance-wide and was not in the list.
- The company-scope rule said that it applies to new files that its
`globs` cannot match.
- The `honeycomb-tracing` skill pointed to `AGENTS.md` for the Telemetry
workflow. That text is now in `rules/data-paths.mdc`.

**Suggested fix**

Limit each rule to what is true, and repoint the skill.

## What Changed

- The API route rule and `AGENTS.md` section 8 now apply to routes that
read or write company data.
- The API route rule names the instance-level routes and says not to add
a company check to them.
- The status code list is now a list of common codes. Reviewers flag an
incorrect code, not an unlisted code.
- The company-scope rule lists the sidebar company order as an
instance-wide exception, in the text, the `globs`, and the Greptile
scope.
- The company-scope rule says "known data" and no longer claims to match
new files.
- The `honeycomb-tracing` skill points to `rules/data-paths.mdc`.

## Verification

- Read the two rules and confirm that each named file exists. A local
script checked the `globs` and the route names.
- Confirm that `.greptile/files.json` is valid JSON and that its scope
matches the rule `globs`.
- No source code changed. No test suite ran locally.

## Risks

- Low risk. The change is documentation only.
- The list of instance-level routes can go out of date when a route is
added.

## Model Used

- Claude (Anthropic), in Claude Code with tool use.
- `claude-fable-5-2`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:19:31 -06:00
DottaandClaude 84007892e9 refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` is the instruction file that each human and AI
contributor reads before a change
> - Pull request #20557 shortened the file, but several sections still
held detail that applies to only some changes
> - The scoped-rules section told agents to read each matching rule in
full, and most changes match five rules
> - The core engineering rules mixed three product invariants with
process conventions
> - This pull request moves that detail to the docs and to scoped rules,
and adds a section on how to fix a bug
> - The benefit is a 211-line file that states what applies to every
change and points to the rest

## Linked Issues or Issue Description

Refs #20557

**Issue type**

Outdated or unclear contributor documentation.

**Where is the issue?**

`AGENTS.md` at the repository root, and `rules/`.

**What's wrong?**

- The scoped-rules section made agents read the full body of each
matching rule before a change. The rule bodies total about 47 KB.
- Sections 4, 6, 8, 9 and 10 repeated detail that belongs in
`doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and
`CONTRIBUTING.md`.
- Section 4 had a database reset command for `data/pglite`. That
directory does not exist.
- Section 5 had seven rules. Four of them were process conventions or
path-specific detail.
- The file did not say how to fix a bug.

**Suggested fix**

Make the rule headers the index. Move section detail to the document or
rule that owns it. Keep only the three invariants in section 5. Add a
bug-fixing section.

## What Changed

- The scoped-rules section now tells agents to list the rule headers
with one `grep` command. Agents read a rule body only for a design
decision in its area, when the description is not sufficient, or when a
reviewer cites it.
- `rules/README.md` says that each `description` must state the required
behavior in one sentence.
- `rules/pull-request-review.mdc` has a description that states its
requirements.
- Section 4 (dev setup) and section 11 (pull requests) now point to
`doc/DEVELOPING.md` and `CONTRIBUTING.md`.
- The schema change workflow moved from section 6 to `doc/DATABASE.md`.
- The API and auth expectations moved from section 8 to the new
`rules/api-route-expectations.mdc`.
- Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer
tells contributors to run `pnpm check:token-gates`.
- Added section 10, "Fixing a Bug": reproduce the error, write a failing
test, write the fix, see the test pass, and verify the fix.
- Section 5 now has three rules: company scope, contract
synchronization, and the control-plane invariants.
- The company-scope exceptions moved to the new
`rules/company-scope-exceptions.mdc`.
- Removed the strategic-docs rule. The plan-doc convention moved to the
repo map. The artifact rule moved to section 7.
- Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now
cite `rules/data-paths.mdc`.
- Regenerated the SDK public source pin, because one of those comments
is in `packages/shared/src/crash-points.ts`.
- Added both new rules to `.greptile/files.json`.
- The API route rule applies only to routes that read or write company
data. Its status code list is a list of common codes.
- The company-scope rule lists the sidebar company order as an
instance-wide exception.
- The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for
the Telemetry workflow.

## Verification

- Open `AGENTS.md` and confirm that each link resolves. A local script
checked this.
- Run `grep -E '^(description|globs|alwaysApply):' rules/*.mdc` and
confirm that each rule prints a description.
- Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD>
--check`. It passes locally.
- `packages/shared/src/crash-points.test.ts` passed locally before the
rebase onto `master`.
- Five source files changed in comments only. No other test suite ran
locally.

## Risks

- An agent that reads only rule descriptions can miss a detail in a rule
body. Reviewers still apply the full rules.
- Pull requests and the definition of done are now sections 11 and 12.
Sections 1 to 9 keep their numbers.
- Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`.
They are historical records.
- Pull request #20565 also changes `AGENTS.md`. The second one to merge
needs a conflict fix in section 1 and section 2.

## Model Used

- Claude (Anthropic), in Claude Code with tool use.
- `claude-fable-5-2`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:00:31 -06:00