mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Background queues wake from work signals. A transaction calls
`signalDatabaseWork(topic, hint)`, and the delivery coordinator wakes
the queue for that topic with that hint.
> - On a hosted instance with zero runs, the stranded-assignment
recovery queue re-checks tasks about 7 times a second. About 2.6 hinting
commits per second drive it. HTTP traffic, chat review polls and local
fixtures with all heartbeat workers do not explain the writer.
> - Statement statistics show what runs, but not who signalled it.
Nothing records the caller of a work signal or why a queue woke.
> - This pull request adds an opt-in attribution log for work signals
and recovery-queue wakes, off by default.
> - The benefit is that one short run on an affected instance names the
writer of any idle signal loop, without code changes.
## Linked Issues or Issue Description
No issue exists. This describes the gap with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
Observability of `signalDatabaseWork`
(`packages/db/src/work-signals.ts`) and of the issue recovery queues
(`server/src/services/recovery/work.ts`).
**Current behavior**
A queue that wakes several times a second at idle shows up only as
repeated statements. Nothing tells you which caller published the hint,
whether the wake came from another process, a reconnect, a direct
settled publish or a stored deadline, or which load and check it caused.
**Proposed behavior**
An environment variable, `PAPERCLIP_DEBUG_WORK_SIGNALS`, read once when
the module loads. It holds a comma-separated topic allowlist, and each
topic can carry `:N` for its stack sampling rate. When it is unset or
empty, the signal path only checks one boolean and does nothing else.
**Reason and benefit**
Several idle-load loops this week took hours of statement analysis
before the writer was found. Attribution by caller turns that into one
log query.
## What Changed
- `packages/db/src/work-signal-debug.ts` (new):
- `parseWorkSignalDebugSpec` reads `topic[:N],…`. An optional numeric
suffix is parsed from the right, so topics that contain colons (for
example `live:<companyId>`) work with or without `:N`. N defaults to 20;
malformed entries are ignored.
- `createWorkSignalDebug` returns `enabled`, `allows(topic)`,
`signal(topic, hint, entry)`, `event(topic, name, fields, stack, entry)`
and `flush()`. A disabled instance constructs no `Error`, captures no
stack, builds no line and starts no timer.
- Stack capture (`captureWorkSignalStack(entry)`) raises
`Error.stackTraceLimit` to 40 and restores it in a `finally`. It calls
`Error.captureStackTrace(holder, entry)`, so V8 cuts every frame above
the public entry (`signalDatabaseWork`, `publishDatabaseWorkSettled`, or
the recovery `wake()`).
- Frames are dropped by file (`node_modules`, `node:internal`, the
signal wrapper files) and by function name (`signalDatabaseWork`,
`notifyDeliveryWork`, …), so trimming works on the bundled `index.mjs`.
Positions are kept raw (`index.mjs:L:C`) for offline source-map lookup.
At most 10 frames are kept.
- Each (topic, event) pair has its own budget of 10 lines per second,
checked BEFORE any stack is captured. A `suppressed` line names the
topic, the dropped event and the count. Frequent `check` lines therefore
never drop `signal` or `direct_wake` lines.
- Stack sampling (1 in N) counts per (topic, event), so `check` and
`load` do not shift the phase of `signal` sampling. `direct_wake` always
captures its caller's stack.
- Lines go to an in-memory buffer that an unref'd 1-second timer flushes
to stderr. The signal path does no I/O and no await.
- `workSignalDebug` is the process instance, built from
`process.env.PAPERCLIP_DEBUG_WORK_SIGNALS` once at module load.
- `packages/db/src/work-signals.ts`: when enabled, it records:
- `signal` in `signalDatabaseWork`;
- `remote_notification` and `notification_reconnect`;
- `settled_publish` (with a sampled stack) for
`publishDatabaseWorkSettled`;
- `uncertain_settled` when an uncertain transaction resolves.
- `server/src/services/recovery/work.ts`: under the same flag:
- each task-queue `load` logs its scope key;
- each `check` logs the issue and company id, `dueAt`, and `wake:
"deadline"` or `"load"`. The marker rides on the queued entry itself,
and only while the flag is on, so nothing is tracked otherwise and
nothing outlives the entry;
- `wake()` logs ONE `direct_wake` line listing the queues it actually
invalidates (assigned-work only when registered), under the first
allowlisted one, with its caller's stack.
- **No request route is logged.** The server has no request async
context to read one synchronously, and this change does not add one.
- `docs/deploy/environment-variables.md`: documents the flag, internal
ids only, raw positions, and that enabling or changing it requires a
restart.
- Tests:
- `packages/db/src/work-signal-debug.test.ts`, 12 tests:
- strict no-op when unset or empty (no `Error`, no `captureStackTrace`,
no stack capture, no sink);
- the process instance is disabled without the env;
- allowlist filtering, and spec parsing including colon topics;
- per-(topic, event) sampling, and always-on direct-wake stacks;
- source-tree trimming, and a synthetic single-file bundle stack with
raw positions kept and wrappers dropped by name;
- real capture below the entry with the limit raised and restored;
- checks at about 8/s alongside signals at about 3/s for 5 seconds, with
every signal line kept;
- suppressed records never capture a stack;
- buffered, timer-driven output.
- `packages/db/src/work-signals.test.ts`: on real PostgreSQL, the signal
and settled-publish paths never call the debug log while the flag is
off.
- `server/src/__tests__/issue-recovery-work-debug-attribution.test.ts`,
on the recovery fixture with real PostgreSQL:
- no recovery log calls while the flag is off;
- with the flag on, a run graph hint logs its signal, the graph-scoped
load and the check;
- `wake()` logs exactly one `direct_wake` that lists the queues and
carries the caller's stack.
## Verification
- `packages/db`: `vitest run src/work-signal-debug.test.ts
src/work-signals.test.ts`: 39 passed. `tsc --noEmit` passes.
- `server`: `vitest run
src/__tests__/issue-recovery-work-debug-attribution.test.ts
src/__tests__/issue-recovery-deadlines.test.ts`: 17 passed. `tsc
--noEmit` passes.
- Staging values:
`PAPERCLIP_DEBUG_WORK_SIGNALS=stranded-assignments:10`, then restart.
- Sample output from the synthetic single-file bundle test:
```
{"msg":"paperclip work signal debug","at":1000000,"event":"signal","topic":"stranded-assignments","hint":"{\"companyId\":\"c\",\"graphRunId\":\"r\"}","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","at async notifyRunDispatch (file:///app/server/dist-runtime/index.mjs:120501:3)","at async PostgresJsDatabase.connection.transaction [as transaction] (file:///app/server/dist-runtime/index.mjs:4021:22)","at async Object.settleRunProjection (file:///app/server/dist-runtime/index.mjs:300777:5)","at async file:///app/server/dist-runtime/index.mjs:310002:17"]}
{"msg":"paperclip work signal debug","at":1000000,"event":"check","topic":"stranded-assignments","issueId":"i","companyId":"c","wake":"load","dueAt":1}
{"msg":"paperclip work signal debug","at":1000000,"event":"direct_wake","topic":"stranded-assignments","queues":"stranded-assignments,orphan-blockers","stack":["at notifyIssueGraphRecovery (file:///app/server/dist-runtime/index.mjs:120456:11)","…"]}
{"msg":"paperclip work signal debug","at":1001000,"event":"suppressed","topic":"stranded-assignments","suppressedEvent":"check","count":17}
```
## Risks
- Low risk. It is off by default. While off, each signal costs one
property read and one boolean check.
- While on, the log can show internal ids from hints (company, issue,
run). It holds no secrets and no user identifiers, and the topic
allowlist limits what is logged.
- Stack capture costs CPU. Sampling (default 1 in 20) and the 10 lines
per second cap bound it.
- The file owner of `packages/db/src/work-signals.ts` reviews this
before merge.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21621.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
140 lines
5.6 KiB
TypeScript
140 lines
5.6 KiB
TypeScript
export type WorkSignalDebugOptions = {
|
|
spec?: string;
|
|
defaultSampleEvery?: number;
|
|
maxLinesPerSecond?: number;
|
|
flushIntervalMs?: number;
|
|
sink?: (text: string) => void;
|
|
now?: () => number;
|
|
captureStack?: (entry: Function) => string | undefined;
|
|
};
|
|
|
|
export type WorkSignalDebugFields = Record<string, string | number | boolean | null | undefined>;
|
|
export type WorkSignalDebugStack = "none" | "sampled" | "always";
|
|
|
|
const DROPPED_FILES = /node_modules|node:internal|work-signals\.(ts|js)|work-signal-debug\.(ts|js)|delivery-work-notifications\.(ts|js)/;
|
|
const DROPPED_FUNCTIONS = new Set(["signalDatabaseWork", "publishDatabaseWorkSettled", "notifyDeliveryWork",
|
|
"captureWorkSignalStack", "processTicksAndRejections"]);
|
|
const STACK_LIMIT = 40;
|
|
|
|
export function parseWorkSignalDebugSpec(spec: string | undefined, defaultSampleEvery = 20): Map<string, number> {
|
|
const topics = new Map<string, number>();
|
|
for (const part of (spec ?? "").split(",")) {
|
|
const match = /^(.+?)(?::(\d+))?$/.exec(part.trim());
|
|
const topic = match?.[1]?.trim();
|
|
if (!topic || !/^[a-z0-9:_-]{1,200}$/i.test(topic) || topic.startsWith(":") || topic.endsWith(":")) continue;
|
|
const every = Number(match?.[2]);
|
|
topics.set(topic, Number.isInteger(every) && every >= 1 ? every : defaultSampleEvery);
|
|
}
|
|
return topics;
|
|
}
|
|
|
|
function frameFunction(line: string) {
|
|
const match = /^at (?:async )?(?:new )?([^\s(]+)(?: \[as [^\]]+\])? \(/.exec(line);
|
|
if (!match) return null;
|
|
const name = match[1]!;
|
|
return name.slice(name.lastIndexOf(".") + 1);
|
|
}
|
|
|
|
export function trimWorkSignalStack(stack: string | undefined, frames = 10): string[] {
|
|
if (!stack) return [];
|
|
return stack.split("\n").slice(1).map(line => line.trim())
|
|
.filter(line => line.startsWith("at ") && !DROPPED_FILES.test(line) && !DROPPED_FUNCTIONS.has(frameFunction(line) ?? ""))
|
|
.slice(0, frames);
|
|
}
|
|
|
|
export function captureWorkSignalStack(entry: Function): string | undefined {
|
|
const holder: { stack?: string } = {};
|
|
const limit = Error.stackTraceLimit;
|
|
try {
|
|
Error.stackTraceLimit = STACK_LIMIT;
|
|
Error.captureStackTrace(holder, entry);
|
|
} finally {
|
|
Error.stackTraceLimit = limit;
|
|
}
|
|
return holder.stack;
|
|
}
|
|
|
|
export function createWorkSignalDebug(options: WorkSignalDebugOptions = {}) {
|
|
const topics = parseWorkSignalDebugSpec(options.spec, options.defaultSampleEvery);
|
|
const enabled = topics.size > 0;
|
|
const now = options.now ?? Date.now;
|
|
const maxLines = options.maxLinesPerSecond ?? 10;
|
|
const sink = options.sink ?? ((text: string) => { process.stderr.write(text); });
|
|
const captureStack = options.captureStack ?? captureWorkSignalStack;
|
|
const counts = new Map<string, number>();
|
|
const budgets = new Map<string, { windowStart: number; lines: number; suppressed: number; topic: string; event: string }>();
|
|
const pending: string[] = [];
|
|
let timer: ReturnType<typeof setInterval> | null = null;
|
|
|
|
function admit(topic: string, event: string) {
|
|
const at = now();
|
|
const key = `${topic}\u0000${event}`;
|
|
let budget = budgets.get(key);
|
|
if (!budget) budgets.set(key, budget = { windowStart: at, lines: 0, suppressed: 0, topic, event });
|
|
if (at - budget.windowStart >= 1000) { budget.windowStart = at; budget.lines = 0; }
|
|
schedule();
|
|
if (budget.lines >= maxLines) { budget.suppressed += 1; return false; }
|
|
budget.lines += 1;
|
|
return true;
|
|
}
|
|
|
|
function emit(topic: string, event: string, entry: Record<string, unknown>) {
|
|
pending.push(JSON.stringify({ msg: "paperclip work signal debug", at: now(), event, topic, ...entry }));
|
|
}
|
|
|
|
function schedule() {
|
|
if (timer || options.flushIntervalMs === 0) return;
|
|
timer = setInterval(flush, options.flushIntervalMs ?? 1000);
|
|
timer.unref?.();
|
|
}
|
|
|
|
function sampled(topic: string, event: string) {
|
|
const key = `${topic}\u0000${event}`;
|
|
const count = counts.get(key) ?? 0;
|
|
counts.set(key, count + 1);
|
|
return count % (topics.get(topic) ?? 1) === 0;
|
|
}
|
|
|
|
function stackFor(topic: string, event: string, mode: WorkSignalDebugStack, entry: Function) {
|
|
if (mode === "none" || (mode === "sampled" && !sampled(topic, event))) return undefined;
|
|
return trimWorkSignalStack(captureStack(entry));
|
|
}
|
|
|
|
function flush() {
|
|
for (const budget of budgets.values()) {
|
|
if (budget.suppressed === 0) continue;
|
|
pending.push(JSON.stringify({ msg: "paperclip work signal debug", at: now(), event: "suppressed",
|
|
topic: budget.topic, suppressedEvent: budget.event, count: budget.suppressed }));
|
|
budget.suppressed = 0;
|
|
}
|
|
if (pending.length === 0) {
|
|
if (timer) { clearInterval(timer); timer = null; }
|
|
return;
|
|
}
|
|
sink(pending.splice(0).join("\n") + "\n");
|
|
}
|
|
|
|
function signal(topic: string, hint: string | undefined, entry: Function = signal) {
|
|
if (!enabled || !topics.has(topic) || !admit(topic, "signal")) return;
|
|
const stack = stackFor(topic, "signal", "sampled", entry);
|
|
emit(topic, "signal", { ...(hint === undefined ? {} : { hint }), ...(stack ? { stack } : {}) });
|
|
}
|
|
|
|
function event(topic: string, name: string, fields: WorkSignalDebugFields = {},
|
|
stack: WorkSignalDebugStack = "none", entry: Function = event) {
|
|
if (!enabled || !topics.has(topic) || !admit(topic, name)) return;
|
|
const frames = stackFor(topic, name, stack, entry);
|
|
emit(topic, name, { ...fields, ...(frames ? { stack: frames } : {}) });
|
|
}
|
|
|
|
return {
|
|
get enabled() { return enabled; },
|
|
allows(topic: string) { return enabled && topics.has(topic); },
|
|
signal,
|
|
event,
|
|
flush,
|
|
};
|
|
}
|
|
|
|
export const workSignalDebug = createWorkSignalDebug({ spec: process.env.PAPERCLIP_DEBUG_WORK_SIGNALS });
|