Files
PaperClipAI/packages/db/src/work-signal-debug.test.ts
T
DottaandClaude Opus 5.5 95a6ab31e6 feat(db): attribute work signals by caller behind a debug flag (#21640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.

## Linked Issues or Issue Description

No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).

**Current behavior**

A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.

**Proposed behavior**

An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.

**Reason and benefit**

Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.

## What Changed

- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
  - `signal` in `signalDatabaseWork`;
  - `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
  - `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
  - each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
  - `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
    - the process instance is disabled without the env;
    - allowlist filtering, and spec parsing including colon topics;
    - per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
    - real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
    - suppressed records never capture a stack;
    - buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
    - no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.

## Verification

- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:

```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```

## Risks

- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #21621.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 13:54:00 -06:00

188 lines
9.7 KiB
TypeScript

import { appendFileSync } from "node:fs";
import { afterEach, describe, expect, it, vi } from "vitest";
import { captureWorkSignalStack, createWorkSignalDebug, parseWorkSignalDebugSpec, trimWorkSignalStack, workSignalDebug } from "./work-signal-debug.js";
const SOURCE_STACK = [
"Error",
" at notifyDeliveryWork (/app/server/src/services/delivery-work-notifications.ts:169:3)",
" at notifyIssueGraphRecovery (/app/server/src/services/issue-recovery-notifications.ts:44:5)",
" at process.processTicksAndRejections (node:internal/process/task_queues:104:5)",
" at someLibrary (/app/node_modules/postgres/src/index.js:1:1)",
" at writer (/app/server/src/services/writer.ts:12:7)",
].join("\n");
const BUNDLE = "file:///app/server/dist-runtime/index.mjs";
const BUNDLED_STACK = [
"Error",
` at notifyDeliveryWork (${BUNDLE}:91234:9)`,
` at notifyIssueGraphRecovery (${BUNDLE}:120456:11)`,
` at async notifyRunDispatch (${BUNDLE}:120501:3)`,
` at async PostgresJsDatabase.connection.transaction [as transaction] (${BUNDLE}:4021:22)`,
` at async Object.settleRunProjection (${BUNDLE}:300777:5)`,
` at async ${BUNDLE}:310002:17`,
" at process.processTicksAndRejections (node:internal/process/task_queues:104:5)",
].join("\n");
function harness(spec: string | undefined, options: { maxLinesPerSecond?: number; stack?: string } = {}) {
let clock = 1_000_000;
const lines: string[] = [];
const sink = vi.fn((text: string) => { lines.push(...text.trim().split("\n")); });
const captureStack = vi.fn((_entry: Function) => options.stack ?? SOURCE_STACK);
const debug = createWorkSignalDebug({ spec, sink, captureStack, now: () => clock, flushIntervalMs: 0, ...options });
return { debug, sink, captureStack, lines, entries: () => lines.map(line => JSON.parse(line)),
advance: (ms: number) => { clock += ms; } };
}
afterEach(() => { vi.restoreAllMocks(); });
describe("work signal debug attribution", () => {
it("is a strict no-op when the flag is unset or empty", () => {
for (const spec of [undefined, "", " , "]) {
const { debug, sink, captureStack } = harness(spec);
const errors = vi.spyOn(globalThis, "Error");
const capture = vi.spyOn(Error, "captureStackTrace");
expect(debug.enabled).toBe(false);
for (let index = 0; index < 50; index++) {
debug.signal("stranded-assignments", "{\"companyId\":\"c\"}");
debug.event("stranded-assignments", "direct_wake", { queues: "a" }, "always");
}
debug.flush();
expect(captureStack).not.toHaveBeenCalled();
expect(errors).not.toHaveBeenCalled();
expect(capture).not.toHaveBeenCalled();
expect(sink).not.toHaveBeenCalled();
}
});
it("leaves the process-wide instance disabled when the environment does not set it", () => {
expect(process.env.PAPERCLIP_DEBUG_WORK_SIGNALS ?? "").toBe("");
expect(workSignalDebug.enabled).toBe(false);
});
it("logs only allowlisted topics", () => {
const { debug, entries } = harness("stranded-assignments,run-dispatch:5");
debug.signal("stranded-assignments", "{\"companyId\":\"c\",\"graphRunId\":\"r\"}");
debug.signal("issue-monitors", "{\"companyId\":\"c\"}");
debug.event("run-dispatch", "load", { scope: "*" });
debug.event("orphan-blockers", "load", { scope: "*" });
debug.flush();
expect(entries().map(entry => [entry.event, entry.topic])).toEqual([
["signal", "stranded-assignments"], ["load", "run-dispatch"]]);
expect(entries()[0].hint).toBe("{\"companyId\":\"c\",\"graphRunId\":\"r\"}");
expect(debug.allows("run-dispatch")).toBe(true);
expect(debug.allows("orphan-blockers")).toBe(false);
});
it("parses per-topic sampling and ignores malformed entries", () => {
expect([...parseWorkSignalDebugSpec("stranded-assignments:3, run-dispatch,bad topic!, :4,x:0,live:7f3e-11,live:7f3e-12:4")])
.toEqual([["stranded-assignments", 3], ["run-dispatch", 20], ["x", 20], ["live:7f3e-11", 20], ["live:7f3e-12", 4]]);
const { debug, entries } = harness("live:7f3e-11");
debug.event("live:7f3e-11", "remote_notification");
debug.flush();
expect(entries()).toEqual([expect.objectContaining({ topic: "live:7f3e-11", event: "remote_notification" })]);
});
it("samples stacks one in N per topic and event, independent of other events", () => {
const { debug, entries, captureStack } = harness("stranded-assignments:3");
for (let index = 0; index < 7; index++) {
debug.signal("stranded-assignments", undefined);
debug.event("stranded-assignments", "check", { issueId: `i${index}` }, "sampled");
debug.event("stranded-assignments", "load", { scope: "*" }, "sampled");
}
debug.flush();
const signals = entries().filter(entry => entry.event === "signal");
expect(signals.map(entry => Boolean(entry.stack))).toEqual([true, false, false, true, false, false, true]);
expect(captureStack).toHaveBeenCalledTimes(9);
});
it("captures an always-on stack for direct wakes on every call", () => {
const { debug, entries, captureStack } = harness("stranded-assignments:20");
for (let index = 0; index < 3; index++) debug.event("stranded-assignments", "direct_wake", { queues: "a,b" }, "always");
debug.flush();
expect(captureStack).toHaveBeenCalledTimes(3);
expect(entries().every(entry => Array.isArray(entry.stack) && entry.queues === "a,b")).toBe(true);
});
it("trims a source-tree stack by file and by function name", () => {
expect(trimWorkSignalStack(SOURCE_STACK)).toEqual([
"at notifyIssueGraphRecovery (/app/server/src/services/issue-recovery-notifications.ts:44:5)",
"at writer (/app/server/src/services/writer.ts:12:7)",
]);
const deep = ["Error", ...Array.from({ length: 30 }, (_, index) => ` at f${index} (/app/server/src/f.ts:${index}:1)`)].join("\n");
expect(trimWorkSignalStack(deep)).toHaveLength(10);
});
it("keeps raw bundle positions and drops wrappers by function name when every frame is in one bundled file", () => {
const { debug, entries } = harness("stranded-assignments:1", { stack: BUNDLED_STACK });
debug.signal("stranded-assignments", "{\"companyId\":\"c\",\"graphRunId\":\"r\"}");
debug.event("stranded-assignments", "check", { issueId: "i", companyId: "c", wake: "load", dueAt: 1 });
debug.event("stranded-assignments", "direct_wake", { queues: "stranded-assignments,orphan-blockers" }, "always");
debug.flush();
expect(entries()[0].stack).toEqual([
`at notifyIssueGraphRecovery (${BUNDLE}:120456:11)`,
`at async notifyRunDispatch (${BUNDLE}:120501:3)`,
`at async PostgresJsDatabase.connection.transaction [as transaction] (${BUNDLE}:4021:22)`,
`at async Object.settleRunProjection (${BUNDLE}:300777:5)`,
`at async ${BUNDLE}:310002:17`,
]);
if (process.env.WORK_SIGNAL_DEBUG_SAMPLE_OUT) appendFileSync(process.env.WORK_SIGNAL_DEBUG_SAMPLE_OUT,
entries().map(entry => JSON.stringify(entry)).join("\n") + "\n");
});
it("captures frames below the entry function with a raised and restored stack limit", () => {
const before = Error.stackTraceLimit;
function signalEntry() { return captureWorkSignalStack(signalEntry); }
function nest(depth: number): string | undefined { return depth === 0 ? signalEntry() : nest(depth - 1); }
const stack = nest(25);
expect(Error.stackTraceLimit).toBe(before);
const frames = (stack ?? "").split("\n").slice(1);
expect(frames.some(frame => frame.includes("signalEntry"))).toBe(false);
expect(frames.filter(frame => frame.includes("nest")).length).toBeGreaterThan(before);
});
it("budgets each topic and event separately so frequent checks never drop signals", () => {
const { debug, entries, advance } = harness("stranded-assignments:1000", { maxLinesPerSecond: 10 });
let signals = 0;
for (let second = 0; second < 5; second++) {
for (let tick = 0; tick < 24; tick++) {
debug.event("stranded-assignments", "check", { issueId: `i${tick}` });
if (tick % 8 === 0) { debug.signal("stranded-assignments", `h${second}-${tick}`); signals++; }
if (tick % 3 === 0) debug.event("stranded-assignments", "check", { issueId: `x${tick}` });
advance(1000 / 24);
}
}
debug.flush();
expect(entries().filter(entry => entry.event === "signal")).toHaveLength(signals);
expect(entries().filter(entry => entry.event === "check").length).toBeLessThanOrEqual(5 * 10 + 10);
const suppressed = entries().filter(entry => entry.event === "suppressed");
expect(suppressed).toEqual([expect.objectContaining({ topic: "stranded-assignments", suppressedEvent: "check" })]);
expect(suppressed[0].count).toBeGreaterThan(0);
});
it("captures no stack for a record the cap suppresses", () => {
const { debug, captureStack, entries } = harness("stranded-assignments:1", { maxLinesPerSecond: 3 });
for (let index = 0; index < 20; index++) {
debug.signal("stranded-assignments", undefined);
debug.event("stranded-assignments", "direct_wake", { queues: "a" }, "always");
}
debug.flush();
expect(captureStack).toHaveBeenCalledTimes(6);
expect(entries().filter(entry => entry.event === "suppressed").map(entry => [entry.suppressedEvent, entry.count]))
.toEqual([["signal", 17], ["direct_wake", 17]]);
});
it("writes nothing on the signal path and flushes later from a timer", () => {
vi.useFakeTimers();
try {
const sink = vi.fn();
const debug = createWorkSignalDebug({ spec: "stranded-assignments", sink, captureStack: () => SOURCE_STACK });
debug.signal("stranded-assignments", undefined);
expect(sink).not.toHaveBeenCalled();
vi.advanceTimersByTime(1000);
expect(sink).toHaveBeenCalledTimes(1);
vi.advanceTimersByTime(5000);
expect(sink).toHaveBeenCalledTimes(1);
} finally { vi.useRealTimers(); }
});
});