mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
feat(observability): span the unnamed gaps in run preparation (#20396)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a task reaches the agent, the server prepares the run: it reads the task, the workspace and the configuration, and then hands the run to the native controller > - With an OTLP endpoint, each preparation step emits a `heartbeat.prepare.<phase>` or `native.controller.<phase>` span > - On a hosted test deployment, 0.3 to 0.5 s of each `task.prepare` sits between those spans, and about 0.5 s more sits between `execution.dispatch_admission` and `native.coordinator.claim` > - Nobody can say which step is slow in that time, so latency work in this area is guesswork > - This pull request adds spans around the existing awaited work in those gaps, with the existing timing helpers and naming conventions > - The benefit is that a trace of a slow preparation now names the step that used the time ## Linked Issues or Issue Description **What existing behavior does this improve?** The OpenTelemetry trace of task preparation. Today it has unnamed gaps between the named preparation spans. **Subsystem affected** Server: heartbeat run preparation and the native controller (`server/src/services/heartbeat.ts`, `server/src/services/native-runtime/native-session-executor.ts`). **Current behavior** On a hosted test deployment (8 runs per group), `task.prepare` takes 2.6 to 3.4 s. Between `heartbeat.prepare.runtime_state` and `heartbeat.prepare.run_identity`, about 0.25 s has no span. Between `execution.dispatch_admission` and `native.coordinator.claim`, about 0.5 s has no span, and in one pair of runs it was 1.4 s. Nobody can tell which step used that time. **Proposed behavior** Spans name the existing awaited reads and writes in those gaps. The change adds no reads and records no content. The operator endpoint gate and the no-operation behaviour stay. **Reason and benefit** Latency work in this area needs a measurement. With these spans, a trace of a slow preparation names the step that used the time, so a fix can be confirmed or rejected after deployment. **Breaking changes** None. The change adds span names to the closed list. It changes no API, schema or behaviour. **Additional context** The server uses postgres.js, which the auto-instrumentation does not cover, so database spans do not fill these gaps. ## What Changed - `server/src/services/execution-preparation-timing.ts`: add the new closed phase names. - `server/src/services/heartbeat.ts`: add `heartbeat.prepare.*` spans for `chat_retry_admission`, `workspace_selection`, `dependency_readiness`, `issue_checkout`, `instance_settings`, `workspace_context`, `project_context`, `plan_wake_routing`, `routine_environment`, `responsible_user` and `execution_start`. The re-reads of the issue after a checkout use the existing `issue_context` name. - `server/src/services/native-runtime/native-session-executor.ts`: add `native.controller.*` spans for `attachment_staging`, `warm_restore`, `span_persistence` (persistence of the preparation spans to the run log) and `state_migration` before the coordinator claim. - `doc/observability.md`: list the new spans. - `server/src/__tests__/heartbeat-preparation-gap-spans.test.ts`: run a real heartbeat through native preparation with an in-memory span exporter. Check that each new span is emitted and that the new preparation spans are children of `task.execution_attempt`. ## Verification - `pnpm --filter @paperclipai/server typecheck` - `npx vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts` (fails without the source change, passes with it) - `npx vitest run src/services/heartbeat/preparation-timing-context.test.ts src/services/execution-preparation-timing.test.ts src/services/native-runtime/native-session-executor.test.ts src/__tests__/heartbeat-context-compaction-execution.test.ts` ## Risks - Low risk. The change only wraps existing awaited calls. Values and errors pass through the timing helper unchanged, and a checkout conflict still reaches the same `catch`. - Phases of 50 ms or more inside `heartbeat.prepare.*` are also written to the run log, as for the existing phases. This adds a few run-log rows on slow runs. - Without an OTLP endpoint, the helpers keep their current no-operation behaviour. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
09241442ea
commit
d95fdd8d79
5 files changed
+234
-73
No files matched your search
@@ -0,0 +1,138 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { context, trace } from "@opentelemetry/api";
|
||||
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
|
||||
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, authUsers, companyMemberships, createDb, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
|
||||
import type { NativeRunIdentity, NativeSession, NativeSessionBackend, PrpEvent } from "@paperclipai/paperclip-runner";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { closeIdleWarmNativeSessionsForRestart } from "../services/native-runtime/native-session-executor.js";
|
||||
import { nativeRestartFixture, waitForCondition } from "./helpers/native-restart-fixture.js";
|
||||
import { getRecoveryPostgresSupport } from "./helpers/recovery-postgres-fixture.js";
|
||||
|
||||
const support = await getRecoveryPostgresSupport({});
|
||||
|
||||
function completingBackend(db: ReturnType<typeof createDb>): NonNullable<HeartbeatServiceOptions["nativeSessionBackendFactory"]> {
|
||||
return execution => {
|
||||
let identity: NativeRunIdentity;
|
||||
let sourceInstanceId = "";
|
||||
let turn = Promise.withResolvers<string>();
|
||||
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true };
|
||||
const session: NativeSession = {
|
||||
identity: () => identity,
|
||||
capabilities: async () => capabilities,
|
||||
attachRun: async input => { identity = input.identity; },
|
||||
startTurn: async () => {
|
||||
const turnId = randomUUID();
|
||||
turn.resolve(turnId);
|
||||
return { turnId };
|
||||
},
|
||||
async *events() {
|
||||
const turnId = await turn.promise;
|
||||
const event = (sourceSeq: number, eventType: PrpEvent["eventType"], payload: Record<string, unknown>): PrpEvent => ({
|
||||
schema: "paperclip.prp.event.v1", sourceKind: "runner", sourceInstanceId,
|
||||
sourceEventId: `${sourceInstanceId}:${identity.runId}:${sourceSeq}`, sourceSeq,
|
||||
runId: identity.runId, normalizedSessionId: identity.sessionId, turnId,
|
||||
schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(), eventType, payload,
|
||||
});
|
||||
yield event(1, "turn.started", {});
|
||||
yield event(2, "item.completed", { kind: "agentMessage", channel: "final", text: "Done." });
|
||||
turn = Promise.withResolvers<string>();
|
||||
yield event(3, "turn.completed", { status: "completed" });
|
||||
},
|
||||
result: async () => null,
|
||||
snapshot: async () => ({ backendKind: "mock", sessionId: "gap-spans-session", identity,
|
||||
providerSessionId: "gap-spans-provider-session", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }),
|
||||
close: async () => {},
|
||||
};
|
||||
const backend: NativeSessionBackend = {
|
||||
descriptor: async () => {
|
||||
const [run] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, execution.binding.runId));
|
||||
if (!run?.runnerInstanceId) throw new Error("Normal native admission did not persist a runner identity");
|
||||
sourceInstanceId = run.runnerInstanceId;
|
||||
return { kind: "mock", name: "preparation-gap-spans", version: "1", capabilities,
|
||||
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
|
||||
},
|
||||
openSession: async input => { identity = input.identity; return session; },
|
||||
recoverSession: async snapshot => { identity = snapshot.identity; return { recovered: true, session }; },
|
||||
};
|
||||
return backend;
|
||||
};
|
||||
}
|
||||
|
||||
describe.skipIf(!support.supported)("heartbeat preparation gap spans", () => {
|
||||
const fixture = nativeRestartFixture({});
|
||||
|
||||
it("emits spans for the work between the named preparation phases", async () => {
|
||||
const { companyId, agentId, temporary } = fixture.current;
|
||||
const db = createDb(temporary.connectionString, { maxConnections: 4 });
|
||||
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
|
||||
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
|
||||
const manager = new AsyncLocalStorageContextManager().enable();
|
||||
const exporter = new InMemorySpanExporter();
|
||||
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
|
||||
expect(context.setGlobalContextManager(manager)).toBe(true);
|
||||
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
|
||||
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: completingBackend(db) });
|
||||
try {
|
||||
const userId = randomUUID();
|
||||
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
|
||||
createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
|
||||
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex", lifecycleMode: "warm", idleTimeoutMs: 60_000 } })
|
||||
.where(eq(agents.id, agentId));
|
||||
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true, enableNativeRunner: true });
|
||||
const [issue] = await db.insert(issues).values({ companyId, title: "Trace the preparation gaps", status: "in_progress",
|
||||
assigneeAgentId: agentId, responsibleUserId: userId, conversationAgentId: agentId, conversationUserId: userId,
|
||||
conversationState: "active" }).returning();
|
||||
const [message] = await db.insert(issueComments).values({ companyId, issueId: issue!.id,
|
||||
authorType: "user", authorUserId: userId, body: "Start." }).returning();
|
||||
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
|
||||
requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true,
|
||||
payload: { issueId: issue!.id, commentId: message!.id },
|
||||
contextSnapshot: { issueId: issue!.id, taskKey: issue!.id, wakeCommentId: message!.id } });
|
||||
expect(run).not.toBeNull();
|
||||
await waitForCondition("the run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0])
|
||||
.toMatchObject({ status: "succeeded" });
|
||||
|
||||
await provider.forceFlush();
|
||||
const spans = exporter.getFinishedSpans();
|
||||
const names = new Set(spans.map(span => span.name));
|
||||
const preparation = [
|
||||
"heartbeat.prepare.chat_retry_admission",
|
||||
"heartbeat.prepare.workspace_selection",
|
||||
"heartbeat.prepare.dependency_readiness",
|
||||
"heartbeat.prepare.instance_settings",
|
||||
"heartbeat.prepare.workspace_context",
|
||||
"heartbeat.prepare.plan_wake_routing",
|
||||
"heartbeat.prepare.routine_environment",
|
||||
"heartbeat.prepare.responsible_user",
|
||||
"heartbeat.prepare.execution_start",
|
||||
];
|
||||
const controller = [
|
||||
"native.controller.attachment_staging",
|
||||
"native.controller.warm_restore",
|
||||
"native.controller.span_persistence",
|
||||
"native.controller.state_migration",
|
||||
];
|
||||
for (const name of [...preparation, ...controller]) expect(names, name).toContain(name);
|
||||
const attempts = spans.filter(span => span.name === "task.execution_attempt");
|
||||
expect(attempts).toHaveLength(1);
|
||||
for (const span of spans.filter(span => preparation.includes(span.name))) {
|
||||
expect(span.parentSpanContext?.spanId, span.name).toBe(attempts[0]!.spanContext().spanId);
|
||||
}
|
||||
} finally {
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
await closeIdleWarmNativeSessionsForRestart();
|
||||
await db.$client.end();
|
||||
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint);
|
||||
trace.disable();
|
||||
context.disable();
|
||||
await provider.shutdown();
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
Reference in new issue
Block a user