feat(observability): span the unnamed gaps in run preparation (#20396)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a task reaches the agent, the server prepares the run: it
reads the task, the workspace and the configuration, and then hands the
run to the native controller
> - With an OTLP endpoint, each preparation step emits a
`heartbeat.prepare.<phase>` or `native.controller.<phase>` span
> - On a hosted test deployment, 0.3 to 0.5 s of each `task.prepare`
sits between those spans, and about 0.5 s more sits between
`execution.dispatch_admission` and `native.coordinator.claim`
> - Nobody can say which step is slow in that time, so latency work in
this area is guesswork
> - This pull request adds spans around the existing awaited work in
those gaps, with the existing timing helpers and naming conventions
> - The benefit is that a trace of a slow preparation now names the step
that used the time

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The OpenTelemetry trace of task preparation. Today it has unnamed gaps
between the named preparation spans.

**Subsystem affected**
Server: heartbeat run preparation and the native controller
(`server/src/services/heartbeat.ts`,
`server/src/services/native-runtime/native-session-executor.ts`).

**Current behavior**
On a hosted test deployment (8 runs per group), `task.prepare` takes 2.6
to 3.4 s. Between `heartbeat.prepare.runtime_state` and
`heartbeat.prepare.run_identity`, about 0.25 s has no span. Between
`execution.dispatch_admission` and `native.coordinator.claim`, about 0.5
s has no span, and in one pair of runs it was 1.4 s. Nobody can tell
which step used that time.

**Proposed behavior**
Spans name the existing awaited reads and writes in those gaps. The
change adds no reads and records no content. The operator endpoint gate
and the no-operation behaviour stay.

**Reason and benefit**
Latency work in this area needs a measurement. With these spans, a trace
of a slow preparation names the step that used the time, so a fix can be
confirmed or rejected after deployment.

**Breaking changes**
None. The change adds span names to the closed list. It changes no API,
schema or behaviour.

**Additional context**
The server uses postgres.js, which the auto-instrumentation does not
cover, so database spans do not fill these gaps.

## What Changed

- `server/src/services/execution-preparation-timing.ts`: add the new
closed phase names.
- `server/src/services/heartbeat.ts`: add `heartbeat.prepare.*` spans
for `chat_retry_admission`, `workspace_selection`,
`dependency_readiness`, `issue_checkout`, `instance_settings`,
`workspace_context`, `project_context`, `plan_wake_routing`,
`routine_environment`, `responsible_user` and `execution_start`. The
re-reads of the issue after a checkout use the existing `issue_context`
name.
- `server/src/services/native-runtime/native-session-executor.ts`: add
`native.controller.*` spans for `attachment_staging`, `warm_restore`,
`span_persistence` (persistence of the preparation spans to the run log)
and `state_migration` before the coordinator claim.
- `doc/observability.md`: list the new spans.
- `server/src/__tests__/heartbeat-preparation-gap-spans.test.ts`: run a
real heartbeat through native preparation with an in-memory span
exporter. Check that each new span is emitted and that the new
preparation spans are children of `task.execution_attempt`.

## Verification

- `pnpm --filter @paperclipai/server typecheck`
- `npx vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts`
(fails without the source change, passes with it)
- `npx vitest run
src/services/heartbeat/preparation-timing-context.test.ts
src/services/execution-preparation-timing.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/__tests__/heartbeat-context-compaction-execution.test.ts`

## Risks

- Low risk. The change only wraps existing awaited calls. Values and
errors pass through the timing helper unchanged, and a checkout conflict
still reaches the same `catch`.
- Phases of 50 ms or more inside `heartbeat.prepare.*` are also written
to the run log, as for the existing phases. This adds a few run-log rows
on slow runs.
- Without an OTLP endpoint, the helpers keep their current no-operation
behaviour.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed 2026-11-09 15:27:58 -08:00
1 parent 09241442ea
commit d95fdd8d79
5 files changed
+234 -73

No files matched your search

@@ -0,0 +1,138 @@
import { randomUUID } from "node:crypto";
import { context, trace } from "@opentelemetry/api";
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks";
import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace";
import { eq } from "drizzle-orm";
import { agents, authUsers, companyMemberships, createDb, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
import type { NativeRunIdentity, NativeSession, NativeSessionBackend, PrpEvent } from "@paperclipai/paperclip-runner";
import { describe, expect, it, vi } from "vitest";
import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { closeIdleWarmNativeSessionsForRestart } from "../services/native-runtime/native-session-executor.js";
import { nativeRestartFixture, waitForCondition } from "./helpers/native-restart-fixture.js";
import { getRecoveryPostgresSupport } from "./helpers/recovery-postgres-fixture.js";
const support = await getRecoveryPostgresSupport({});
function completingBackend(db: ReturnType<typeof createDb>): NonNullable<HeartbeatServiceOptions["nativeSessionBackendFactory"]> {
return execution => {
let identity: NativeRunIdentity;
let sourceInstanceId = "";
let turn = Promise.withResolvers<string>();
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true };
const session: NativeSession = {
identity: () => identity,
capabilities: async () => capabilities,
attachRun: async input => { identity = input.identity; },
startTurn: async () => {
const turnId = randomUUID();
turn.resolve(turnId);
return { turnId };
},
async *events() {
const turnId = await turn.promise;
const event = (sourceSeq: number, eventType: PrpEvent["eventType"], payload: Record<string, unknown>): PrpEvent => ({
schema: "paperclip.prp.event.v1", sourceKind: "runner", sourceInstanceId,
sourceEventId: `${sourceInstanceId}:${identity.runId}:${sourceSeq}`, sourceSeq,
runId: identity.runId, normalizedSessionId: identity.sessionId, turnId,
schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(), eventType, payload,
});
yield event(1, "turn.started", {});
yield event(2, "item.completed", { kind: "agentMessage", channel: "final", text: "Done." });
turn = Promise.withResolvers<string>();
yield event(3, "turn.completed", { status: "completed" });
},
result: async () => null,
snapshot: async () => ({ backendKind: "mock", sessionId: "gap-spans-session", identity,
providerSessionId: "gap-spans-provider-session", cursor: null, activeTurnId: null, pendingRuntimeRequests: [], lineage: [] }),
close: async () => {},
};
const backend: NativeSessionBackend = {
descriptor: async () => {
const [run] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, execution.binding.runId));
if (!run?.runnerInstanceId) throw new Error("Normal native admission did not persist a runner identity");
sourceInstanceId = run.runnerInstanceId;
return { kind: "mock", name: "preparation-gap-spans", version: "1", capabilities,
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
},
openSession: async input => { identity = input.identity; return session; },
recoverSession: async snapshot => { identity = snapshot.identity; return { recovered: true, session }; },
};
return backend;
};
}
describe.skipIf(!support.supported)("heartbeat preparation gap spans", () => {
const fixture = nativeRestartFixture({});
it("emits spans for the work between the named preparation phases", async () => {
const { companyId, agentId, temporary } = fixture.current;
const db = createDb(temporary.connectionString, { maxConnections: 4 });
const endpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test");
const manager = new AsyncLocalStorageContextManager().enable();
const exporter = new InMemorySpanExporter();
const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter })] });
expect(context.setGlobalContextManager(manager)).toBe(true);
expect(trace.setGlobalTracerProvider(provider)).toBe(true);
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: completingBackend(db) });
try {
const userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex", lifecycleMode: "warm", idleTimeoutMs: 60_000 } })
.where(eq(agents.id, agentId));
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true, enableNativeRunner: true });
const [issue] = await db.insert(issues).values({ companyId, title: "Trace the preparation gaps", status: "in_progress",
assigneeAgentId: agentId, responsibleUserId: userId, conversationAgentId: agentId, conversationUserId: userId,
conversationState: "active" }).returning();
const [message] = await db.insert(issueComments).values({ companyId, issueId: issue!.id,
authorType: "user", authorUserId: userId, body: "Start." }).returning();
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true,
payload: { issueId: issue!.id, commentId: message!.id },
contextSnapshot: { issueId: issue!.id, taskKey: issue!.id, wakeCommentId: message!.id } });
expect(run).not.toBeNull();
await waitForCondition("the run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)))[0])
.toMatchObject({ status: "succeeded" });
await provider.forceFlush();
const spans = exporter.getFinishedSpans();
const names = new Set(spans.map(span => span.name));
const preparation = [
"heartbeat.prepare.chat_retry_admission",
"heartbeat.prepare.workspace_selection",
"heartbeat.prepare.dependency_readiness",
"heartbeat.prepare.instance_settings",
"heartbeat.prepare.workspace_context",
"heartbeat.prepare.plan_wake_routing",
"heartbeat.prepare.routine_environment",
"heartbeat.prepare.responsible_user",
"heartbeat.prepare.execution_start",
];
const controller = [
"native.controller.attachment_staging",
"native.controller.warm_restore",
"native.controller.span_persistence",
"native.controller.state_migration",
];
for (const name of [...preparation, ...controller]) expect(names, name).toContain(name);
const attempts = spans.filter(span => span.name === "task.execution_attempt");
expect(attempts).toHaveLength(1);
for (const span of spans.filter(span => preparation.includes(span.name))) {
expect(span.parentSpanContext?.spanId, span.name).toBe(attempts[0]!.spanContext().spanId);
}
} finally {
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", endpoint);
trace.disable();
context.disable();
await provider.shutdown();
}
}, 60_000);
});