mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
Measure database and projection costs in runtime tool discovery (#22347)
## Thinking Path > - Paperclip starts agent turns with instructions for the connections the agent can use. > - The gateway reads the connected-tool catalog during prompt preparation. > - A single discovery span currently covers both database waits and descriptor construction. > - Maintainers need to distinguish these costs before changing catalog reads. > - This change records the transaction, both queries, availability checks, and descriptor construction. ## Linked Issues or Issue Description **Subsystem affected** — Runtime tool discovery and run preparation. **Problem or motivation** — The existing catalog span cannot explain time spent before a remote turn starts. **Proposed solution** — Measure the existing read in place. Record the number of entries and connections returned. **Alternatives considered** — A local microbenchmark cannot reproduce a hosted database or a deployed catalog's sizes. A change to the query would confound the measurement. **Roadmap alignment** — This adds diagnostic detail to an existing operation. Related work includes #14864, which bounded catalog memory use. **Additional context** — The PostgreSQL reads share one read-only snapshot. ## What Changed - Add a transaction span with children for the entry read and the matched-connection read. - Attach the two returned row counts to that transaction. - Time availability checks and descriptor construction under the existing catalog discovery span. - Document which preparation timings can also reach the native run event table. ## Verification - Ran the gateway listing-memory, discovery HTTP, and execution timing context suites with local PostgreSQL. All 30 tests passed. - Checked the parentage and row counts with a real OpenTelemetry exporter in memory. - Ran the standing-instruction integration suite and measured the catalog from two staging application containers with read-only database sessions. ## Risks Each recorded catalog operation adds up to five spans. The existing per-phase bound applies. Only the OTel transaction span includes the aggregate row counts. With or without OTel, the existing native preparation timing path can also persist a run event for each new phase that takes at least 50 ms, subject to its normal caps. The executor defers those writes. The database query, tool eligibility, and authorization contracts stay the same. ## Model Used OpenAI Codex directed and qualified the work. GPT-6.2 Sol implemented the instrumentation and regression test. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
1 parent
a640b4dac8
commit
28043f8b93
5 files changed
+130
-41
No files matched your search
@@ -1,4 +1,7 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { context, trace } from "@opentelemetry/api";
|
||||
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
|
||||
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
@@ -24,6 +27,7 @@ import {
|
||||
createToolAccessDecisionCache,
|
||||
toolAccessPolicyService,
|
||||
} from "../services/tool-access-policy.js";
|
||||
import { withRunExecutionTiming } from "../services/execution-preparation-timing.js";
|
||||
import { createToolGatewayService } from "../services/tool-gateway.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
@@ -252,6 +256,48 @@ describeEmbeddedPostgres("tool gateway listing memory", () => {
|
||||
expect(connectionReadParams[0]).not.toContain(emptyConnection.id);
|
||||
});
|
||||
|
||||
it("exports the catalog transaction, query and projection boundaries for a gateway listing", async () => {
|
||||
const fixture = await createListingFixture(db, 20, { connectionCount: 2 });
|
||||
const gateway = createToolGatewayService(db);
|
||||
const exporter = new InMemorySpanExporter();
|
||||
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
|
||||
const manager = new AsyncLocalStorageContextManager().enable();
|
||||
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
|
||||
try {
|
||||
expect(context.setGlobalContextManager(manager)).toBe(true);
|
||||
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
|
||||
const tools = await withRunExecutionTiming(fixture.run.id, () => gateway.listToolsForNamedGateway({
|
||||
gatewayId: fixture.namedGateway.id,
|
||||
bearerToken: fixture.token.token,
|
||||
}));
|
||||
expect(tools.filter((tool) => tool.catalogEntryId)).toHaveLength(18);
|
||||
|
||||
const spans = exporter.getFinishedSpans();
|
||||
const prefix = "connectors.discovery.connections";
|
||||
const connectionPhase = spans.find((span) => span.name === prefix)!;
|
||||
const transaction = spans.find((span) => span.name === `${prefix}.transaction`)!;
|
||||
expect(transaction.parentSpanContext?.spanId).toBe(connectionPhase.spanContext().spanId);
|
||||
expect(transaction.attributes).toMatchObject({
|
||||
"paperclip.execution.catalogEntryCount": 20,
|
||||
"paperclip.execution.connectionCount": 2,
|
||||
});
|
||||
for (const name of ["catalog_entries", "matched_connections"]) {
|
||||
const query = spans.find((span) => span.name === `${transaction.name}.${name}`);
|
||||
expect(query?.parentSpanContext?.spanId).toBe(transaction.spanContext().spanId);
|
||||
}
|
||||
for (const name of ["availability", "descriptors"]) {
|
||||
const phase = spans.find((span) => span.name === `${prefix}.${name}`);
|
||||
expect(phase?.parentSpanContext?.spanId).toBe(connectionPhase.spanContext().spanId);
|
||||
}
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
trace.disable();
|
||||
context.disable();
|
||||
manager.disable();
|
||||
await provider.shutdown();
|
||||
}
|
||||
});
|
||||
|
||||
it("never selects the whole run snapshot or result when it decides access", async () => {
|
||||
// Without agent or company bindings, the project binding is the narrowest match.
|
||||
const fixture = await createListingFixture(db, 3, { broadBindings: false });
|
||||
|
||||
Reference in new issue
Block a user