perf(recovery): match scoped deadline entries only against scopes of their own company (#21467)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live-run recovery uses the shared scoped deadline worker
(`scoped-deadline-work.ts`). Each queue keeps its entries (one for each
running run) and the scopes that writes marked dirty (for example one
company and one run)
> - To schedule and to pick due work, the worker tested every entry
against every dirty scope. It also decoded each scope hint with
`JSON.parse` again inside those loops, and each reconstructed scope
walked all entries to drop vanished ones
> - With many runs and many dirty run scopes, this work grows with
entries × scopes on the event loop of the one server process that
controls all runs
> - This pull request lets an owner declare a partition key for its
scopes. Live-run recovery uses the company. The worker then tests an
entry only against the scopes of its own company, and decodes each hint
once
> - The benefit is that matching cost grows with entries + scopes, with
the same results

## Linked Issues or Issue Description

Refs #21444 (the lease renewal change from the same load model).

**What existing behavior does this improve?**
The scoped deadline worker in
`server/src/services/scoped-deadline-work.ts`, which live-run recovery
(`server/src/services/heartbeat/live-recovery-work.ts`) uses for its
seven queues.

**Subsystem affected**
Server: background work scheduling for live-run and resource recovery.

**Current behavior**
For 400 entries in 40 companies and 40 dirty run scopes, one schedule
computation and one pass call `matches` 27,830 times and decode each
hint many times.

**Proposed behavior**
The same case calls `matches` fewer than 3,200 times and decodes each
hint at most once. Owners that do not declare a key behave exactly as
before.

**Reason and benefit**
The plan for 1,000 concurrent agents needs per-run background work that
does not grow with the product of runs and changes. The work runs on the
main thread of the controlling process.

**Breaking changes**
None. `scopeKey` is optional.

## What Changed

- `scoped-deadline-work.ts`: new optional `scopeKey` option
(`entry(entry)` and `scope(scope)`). The contract: a scope with a
defined key matches only entries with the same key. A scope without a
key (for example `*`) is tested against every entry, as before.
- The worker indexes the dirty scopes by key, so `next()` and the due
filter test each entry only against the scopes of its key and the
unkeyed scopes.
- When a reconstructed keyed scope drops vanished entries, it visits
only the entries of its key (an index built once for each pass).
- Scope hints are decoded once and the decoded value is reused while the
hint is dirty or scanning.
- `live-recovery-work.ts`: declares the company as the key. Its
`matches` already requires the same company for every defined scope.

## Verification

- New test in `scoped-deadline-work.test.ts`: 400 entries in 40
companies, then 40 run-scoped invalidations. It requires fewer than
3,200 `matches` calls and at most 40 decodes, and the next deadline is
unchanged.
- Red first: with `origin/master` `scoped-deadline-work.ts`, the test
failed with 27,830 `matches` calls.
- `pnpm exec vitest run src/__tests__/scoped-deadline-work.test.ts`: 22
passed.
- The live-run recovery matrix (`live-recovery-work.test.ts`) is covered
by CI.

## Risks

- If an owner declares a key that its `matches` does not respect, the
worker can miss a match. Only live-run recovery declares a key, and its
`matches` returns false for any defined scope of another company.
- Low risk otherwise: owners without a key keep the old code path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 14:36:21 -06:00
1 parent 754b1f36d3
commit 533d02ff1a
3 files changed
+134 -8

No files matched your search

@@ -624,3 +624,76 @@ it("keeps partial reconstruction and failed checks uncertified until they finish
fail = false; await vi.advanceTimersByTimeAsync(30_000);
expect(worker.reconciliationSnapshot().reconciled).toBe(true);
});
it("tests each entry only against the dirty scopes of its own key, and decodes each hint once", async () => {
vi.useFakeTimers();
const onError = vi.fn();
const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) });
coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError });
type Scope = { companyId: string; runId?: string };
type Row = { id: string; companyId: string; dueAt: number };
const dueAt = Date.now() + 10 * 60_000;
const rows: Row[] = Array.from({ length: 40 }, (_, company) => Array.from({ length: 10 }, (_, run) =>
({ id: `run-${company}-${run}`, companyId: `company-${company}`, dueAt }))).flat();
const inScope = (row: Row, scope?: Scope) => !scope
|| row.companyId === scope.companyId && (!scope.runId || row.id === scope.runId);
const matches = vi.fn(inScope);
const decodeScope = vi.fn((hint: string): Scope | undefined => hint === "*" ? undefined : JSON.parse(hint));
const worker = registerScopedDeadlineWork<Row, Scope>(coordinator, {
queue: DELIVERY_QUEUES.activeExecutionRecovery,
minimumPassIntervalMs: 0,
decodeScope,
matches,
scopeKey: { entry: row => row.companyId, scope: scope => scope.companyId },
load: async scope => rows.filter(row => inScope(row, scope)),
check: async row => row,
}, () => true);
await worker.ready;
await vi.advanceTimersByTimeAsync(1);
matches.mockClear(); decodeScope.mockClear();
// One run of each company changes: 40 run-scoped hints at once.
for (let company = 0; company < 40; company++) {
worker.invalidate(JSON.stringify({ companyId: `company-${company}`, runId: `run-${company}-0` }));
}
await vi.advanceTimersByTimeAsync(1);
await vi.advanceTimersByTimeAsync(1);
// Without the key every entry is tested against every dirty scope
// (400 entries x 40 scopes for each schedule computation).
expect(matches.mock.calls.length).toBeLessThan(rows.length * 8);
expect(decodeScope.mock.calls.length).toBeLessThanOrEqual(40);
expect(coordinator.nextWakeAt()).toBe(dueAt);
expect(onError).not.toHaveBeenCalled();
});
it("drops an entry that vanished from a keyed scope and keeps other companies' saved work", async () => {
vi.useFakeTimers();
const onError = vi.fn();
const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) });
coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError });
type Scope = { companyId: string; runId?: string };
type Row = { id: string; companyId: string; dueAt: number };
const dueAt = Date.now() + 60_000;
let rows: Row[] = ["a", "b"].flatMap(company => [1, 2].map(run => ({ id: `${company}-${run}`, companyId: company, dueAt })));
const inScope = (row: Row, scope?: Scope) => !scope
|| row.companyId === scope.companyId && (!scope.runId || row.id === scope.runId);
const check = vi.fn(async (_row: Row) => null);
const worker = registerScopedDeadlineWork<Row, Scope>(coordinator, {
queue: DELIVERY_QUEUES.activeExecutionRecovery,
minimumPassIntervalMs: 0,
decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint),
matches: inScope,
scopeKey: { entry: row => row.companyId, scope: scope => scope.companyId },
load: async scope => rows.filter(row => inScope(row, scope)),
check,
}, () => true);
await worker.ready;
await vi.advanceTimersByTimeAsync(1);
rows = rows.filter(row => row.id !== "a-1");
worker.invalidate(JSON.stringify({ companyId: "a" }));
await vi.advanceTimersByTimeAsync(1);
expect(check).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(60_000);
expect(check.mock.calls.map(([row]) => row.id).sort()).toEqual(["a-2", "b-1", "b-2"]);
expect(coordinator.nextWakeAt()).toBeNull();
expect(onError).not.toHaveBeenCalled();
});