Files
PaperClipAI/server/src/__tests__/decision-queues-routes.test.ts
T
DottaandPaperclip 4e27a82a42 fix(server): wake approval queue seeding from committed changes (#17510)
## Thinking Path

> - Paperclip manages work and human decisions for AI-agent companies.
> - Automatic decision queues store membership for pending attention
sources.
> - The old attention maintenance scan created that membership as a side
effect.
> - Moving approval retention out of the scan also removed approval
queue discovery.
> - Queue API consumers do not build the attention feed first, so they
can miss pending PR approvals.
> - This pull request gives approval queue discovery its own worker and
commit signals.
> - The worker becomes quiet after it saves every missing membership.

## Linked Issues or Issue Description

Refs #17497. Stacked on #17508, following #17505.

This is a correctness dependency for the approval retention migration in
#17497. Do not treat that migration as complete without this
queue-population path. A real PostgreSQL regression found no PR queue
when a pending approval had a linked PR work product and the maintenance
feed excluded approvals.

Related: #13364 proposes pruning stale seeded memberships. This PR
preserves existing membership retention and does not implement pruning.
#15149 documents the decision-queue APIs.

## What Changed

- Add an approval queue worker that selects missing PR-queue memberships
from existing durable records. Bound each pass to 100 rows. Continue
past failed rows and revisit them on retry.
- Reuse the queue service's seed writer through a normalized source
contract. Keep queue rules, uniqueness, and audit writes in that
service. Keep feed readers compatible.
- Wake from committed approval and link changes, issue/company
visibility, PR work-product changes and imports, and queue-rule changes.
Reuse the existing transaction boundaries.
- Recheck approval status under a row lock. Serialize queue-rule changes
with the seed writer. Preserve required recovery during drain and
shutdown.
- Register startup reconstruction under the existing maintenance switch.
Leave other attention-source seeding with its current owner until its
own migration.
- Document the complete producer audit and stack dependency. Add a
validated external PostgreSQL option to the existing queue route suite
so it runs on hosts without embedded database support.

## Verification

Current tested revision: `7948136ef8`. This PR targets master after
#17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved
the complete tested integration tree, before the review repairs below.
Only the original 25 seeding paths differ from master, and database
schema/migration metadata is identical.

- **56 cases pass in the three complete affected suites:** 28 approval
seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and
production build pass.
- Three real PostgreSQL retry regressions failed then passed: new
matching arrivals cannot postpone a recovered failure indefinitely;
successful partial progress resets old failure penalties; an empty
cursor rollover does not reset outage backoff. Exact one- and two-second
retry assertions, eventual complete recovery and existing
capped-backoff/company isolation tests pass.
- Three genuinely empty external databases failed before setup repair
because tables were missing. Each suite now applies migrations itself.
All 55 cases passed on a second set of fresh databases; adding the final
rollover guard then passed all 56 cases using those now-migrated
fixtures. No production migration or table is added.
- The initial expanded fresh run passed 54 cases but failed a cleanup
expectation that omitted a required cursor rollover pass. The fixture
now permits that explicit continuation while preserving exact deadline
and complete-membership assertions. All failure records remain in
`doc/idle-approval-queue-seeding.md`.
- The prior integrated revision `4e725e2055` passed **522 cases in
fifteen complete producer/consumer suites**, plus types/build. Its
unchanged consumer code and historical deadlock, cross-company
amplification, backoff, gateway integration and setup evidence remain
documented; those counts are not relabeled as current acceptance.

Commands: server Vitest for approval queue seeding, queue routes and
task watchdogs; server typecheck/build. Fresh exact-head
Greptile/Ponytail reviews and all CI checks remain required. No full
local repository-suite, deployed suspension or cost claim is made.

## Risks

- Signals cover the supported in-process service writers. Independent
SQL writers and multiple server replicas need a separate notification
contract.
- The source predicate must stay aligned with unscoped attention
maintenance. The first linked issue, company scope, pending status, and
execution visibility are tested. A pending approval itself keeps its
linked external conversation active.
- Existing enabled seed rules can re-add manually removed matching
memberships. This preserves the old scan's behavior. Completed or
archived membership is not pruned.
- Queue seeding and retention are separate owners. Other attention
sources still need their own event-driven discovery before the aggregate
scan can be retired.
- This PR does not extend Cloud wake coverage, relax sleep admission, or
claim measured host/database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:33:01 -05:00

497 lines
19 KiB
TypeScript

import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import express from "express";
import request from "supertest";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
activityLog,
agents,
approvals,
companies,
createDb, applyPendingMigrations,
decisionQueueItems,
decisionQueues,
decisionTriage,
decisionTriageEvents,
issueThreadInteractions,
issueWorkProducts,
issues,
} from "@paperclipai/db";
import type { AttentionItem } from "@paperclipai/shared";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { errorHandler } from "../middleware/index.js";
import { decisionQueueRoutes } from "../routes/decision-queues.js";
import { decisionQueueService } from "../services/decision-queues.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISION_QUEUES_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
console.warn(
`Skipping embedded Postgres decision queue tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
);
}
describeEmbeddedPostgres("decision queue routes", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
if (external) await applyPendingMigrations(external);
else tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decision-queues-");
db = createDb(external ?? tempDb!.connectionString);
}, 30_000);
afterEach(async () => {
await db.delete(decisionTriageEvents);
await db.delete(decisionQueueItems);
await db.delete(decisionTriage);
await db.delete(decisionQueues);
await db.delete(activityLog);
await db.delete(issueThreadInteractions);
await db.delete(issueWorkProducts);
await db.delete(approvals);
await db.delete(issues);
await db.delete(agents);
await db.delete(companies);
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
async function seed() {
const companyId = randomUUID();
const agentId = randomUUID();
const issueId = randomUUID();
const interactionId = randomUUID();
const approvalId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Decision Queue Co",
issuePrefix: "DQC",
});
await db.insert(agents).values({
id: agentId,
companyId,
name: "Prioritizer",
role: "engineer",
status: "idle",
adapterType: "codex_local",
adapterConfig: {},
runtimeConfig: {},
permissions: {},
});
await db.insert(issues).values({
id: issueId,
companyId,
identifier: "DQC-1",
title: "Review the rollout",
status: "in_review",
assigneeAgentId: agentId,
});
await db.insert(issueThreadInteractions).values({
id: interactionId,
companyId,
issueId,
kind: "ask_user_questions",
status: "pending",
payload: { version: 1, questions: [] } as never,
});
await db.insert(approvals).values({
id: approvalId,
companyId,
type: "request_board_approval",
status: "pending",
payload: {},
});
return { companyId, agentId, issueId, interactionId, approvalId };
}
function app(actor: Record<string, unknown>) {
const testApp = express();
testApp.use(express.json());
testApp.use((req, _res, next) => {
(req as any).actor = actor;
next();
});
testApp.use("/api", decisionQueueRoutes(db));
testApp.use(errorHandler);
return testApp;
}
function boardActor(companyId: string, userId = "board-user") {
return {
type: "board",
source: "local_implicit",
userId,
companyIds: [companyId],
isInstanceAdmin: false,
};
}
function agentActor(companyId: string, agentId: string) {
return {
type: "agent",
source: "agent_key",
companyId,
agentId,
keyId: null,
keyScope: { kind: "standard" },
runId: null,
};
}
it("creates idempotently, patches, lists by updated time, and audits queue mutations", async () => {
const { companyId } = await seed();
const board = boardActor(companyId);
const first = await request(app(board)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "launches",
title: "Launches",
description: "Ship decisions",
}).expect(201);
const repeated = await request(app(board)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "launches",
title: "Ignored duplicate title",
}).expect(200);
expect(repeated.body.id).toBe(first.body.id);
expect(repeated.body.title).toBe("Launches");
expect(repeated.body.itemCount).toBe(0);
const patched = await request(app(board))
.patch(`/api/companies/${companyId}/decision-queues/launches`)
.send({ title: "Launch desk", retentionDays: 45 })
.expect(200);
expect(patched.body).toMatchObject({ title: "Launch desk", retentionDays: 45 });
await request(app(board)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "older",
title: "Older",
}).expect(201);
const listed = await request(app(board)).get(`/api/companies/${companyId}/decision-queues`).expect(200);
expect(listed.body.map((queue: { key: string }) => queue.key)).toEqual(["older", "launches"]);
const seedRules = await request(app(board))
.get(`/api/companies/${companyId}/decision-queue-seed-rules`)
.expect(200);
expect(seedRules.body.map((seed: { key: string }) => seed.key)).toEqual(["prs", "plans", "questions"]);
const events = await db.select().from(decisionTriageEvents)
.where(eq(decisionTriageEvents.queueId, first.body.id));
expect(events.map((event) => event.action)).toEqual(["queue.created", "queue.updated"]);
expect(events.every((event) => event.actorUserId === "board-user")).toBe(true);
});
it("adds and removes three attention source kinds and hides board-only membership from agents", async () => {
const { companyId, agentId, issueId, interactionId, approvalId } = await seed();
const board = boardActor(companyId);
await request(app(board)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "triage",
title: "Triage",
}).expect(201);
for (const source of [
{ sourceKind: "approval", sourceId: approvalId },
{ sourceKind: "issue_thread_interaction", sourceId: interactionId },
{ sourceKind: "review", sourceId: issueId },
]) {
await request(app(board))
.post(`/api/companies/${companyId}/decision-queues/triage/items`)
.send(source)
.expect(201);
}
const items = await request(app(board))
.get(`/api/companies/${companyId}/decision-queues/triage/items`)
.expect(200);
expect(items.body).toHaveLength(3);
const repeated = await request(app(board)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "triage",
title: "Ignored duplicate title",
}).expect(200);
expect(repeated.body.itemCount).toBe(3);
const agentList = await request(app(agentActor(companyId, agentId)))
.get(`/api/companies/${companyId}/decision-queues`)
.expect(200);
expect(agentList.body[0].itemCount).toBe(2);
const agentItems = await request(app(agentActor(companyId, agentId)))
.get(`/api/companies/${companyId}/decision-queues/triage/items`)
.expect(200);
expect(agentItems.body.map((item: { sourceKind: string }) => item.sourceKind).sort()).toEqual([
"issue_thread_interaction",
"review",
]);
await request(app(agentActor(companyId, agentId)))
.delete(`/api/companies/${companyId}/decision-queues/triage/items/approval/${approvalId}`)
.expect(404);
await request(app(board))
.delete(`/api/companies/${companyId}/decision-queues/triage/items/review/${issueId}`)
.send({ reason: "Handled in the release review" })
.expect(200);
expect(await db.select().from(decisionQueueItems).where(and(
eq(decisionQueueItems.companyId, companyId),
eq(decisionQueueItems.sourceKind, "review"),
))).toHaveLength(0);
const removalAudit = await db.select().from(activityLog).where(and(
eq(activityLog.companyId, companyId),
eq(activityLog.action, "decision_queue_item.removed"),
)).then((rows) => rows[0]);
expect(removalAudit?.details).toMatchObject({
sourceKind: "review",
sourceId: issueId,
reason: "Handled in the release review",
});
const removalEvent = await db.select().from(decisionTriageEvents).where(and(
eq(decisionTriageEvents.companyId, companyId),
eq(decisionTriageEvents.sourceKind, "review"),
eq(decisionTriageEvents.sourceId, issueId),
eq(decisionTriageEvents.action, "queue_item.removed"),
)).then((rows) => rows[0]);
expect(removalEvent?.details).toEqual({ reason: "Handled in the release review" });
await db.delete(approvals).where(eq(approvals.id, approvalId));
await request(app(board))
.delete(`/api/companies/${companyId}/decision-queues/triage/items/approval/${approvalId}`)
.expect(200);
expect(await db.select().from(decisionQueueItems).where(and(
eq(decisionQueueItems.companyId, companyId),
eq(decisionQueueItems.sourceKind, "approval"),
))).toHaveLength(0);
});
it("materializes data-backed starter queues from plan, question, and pull-request signals", async () => {
const { companyId, issueId, interactionId } = await seed();
await db.insert(issueWorkProducts).values({
companyId,
issueId,
type: "pull_request",
provider: "github",
title: "PR 42",
status: "open",
});
function attentionItem(input: {
sourceKind: AttentionItem["sourceKind"];
sourceId: string;
subjectKind: AttentionItem["subject"]["kind"];
metadata?: Record<string, unknown>;
issueId?: string;
}): AttentionItem {
return {
id: `${input.sourceKind}:${input.sourceId}`,
companyId,
sourceKind: input.sourceKind,
subject: {
kind: input.subjectKind,
id: input.sourceId,
companyId,
title: "Seed candidate",
identifier: null,
status: "pending",
href: null,
metadata: input.metadata,
},
whyNow: "test",
decisionVerbs: [],
inlineResolvable: true,
entryRule: "test",
exitRule: "test",
dedupKey: `${input.sourceKind}:${input.sourceId}`,
dismissalKey: `${input.sourceKind}:${input.sourceId}`,
dismissal: null,
severity: "medium",
rank: 1,
activityAt: new Date().toISOString(),
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
relatedIssue: input.issueId ? {
kind: "issue",
id: input.issueId,
companyId,
title: "Issue",
identifier: "DQC-1",
status: "in_review",
href: null,
} : null,
project: null,
workspace: null,
detail: null,
trainingExampleId: null,
};
}
const planId = randomUUID();
const candidates = [
attentionItem({ sourceKind: "review", sourceId: issueId, subjectKind: "issue", issueId }),
attentionItem({
sourceKind: "issue_thread_interaction",
sourceId: planId,
subjectKind: "interaction",
metadata: { kind: "request_confirmation", isPlanTarget: true, issueId },
issueId,
}),
attentionItem({
sourceKind: "issue_thread_interaction",
sourceId: interactionId,
subjectKind: "interaction",
metadata: { kind: "ask_user_questions", issueId },
issueId,
}),
];
await decisionQueueService(db).materializeSeededQueues(companyId, candidates);
const queues = await db.select().from(decisionQueues).where(eq(decisionQueues.companyId, companyId));
expect(queues.map((queue) => queue.key).sort()).toEqual(["plans", "prs", "questions"]);
expect(queues.every((queue) => queue.seedRulesEnabled && queue.seedRules.length === 1)).toBe(true);
const seededItems = await db.select({ queueId: decisionQueueItems.queueId })
.from(decisionQueueItems)
.where(eq(decisionQueueItems.companyId, companyId));
expect(seededItems).toHaveLength(5);
const prQueue = queues.find((queue) => queue.key === "prs")!;
expect(seededItems.filter((item) => item.queueId === prQueue.id)).toHaveLength(3);
const firstEvents = await db.select().from(decisionTriageEvents)
.where(eq(decisionTriageEvents.companyId, companyId));
const firstActivity = await db.select().from(activityLog).where(eq(activityLog.companyId, companyId));
const firstUpdatedAt = new Map(queues.map((queue) => [queue.key, queue.updatedAt.toISOString()]));
await decisionQueueService(db).materializeSeededQueues(companyId, candidates);
const unchangedQueues = await db.select().from(decisionQueues).where(eq(decisionQueues.companyId, companyId));
expect(unchangedQueues.every((queue) => queue.updatedAt.toISOString() === firstUpdatedAt.get(queue.key))).toBe(true);
expect(await db.select().from(decisionTriageEvents).where(eq(decisionTriageEvents.companyId, companyId)))
.toHaveLength(firstEvents.length);
expect(await db.select().from(activityLog).where(eq(activityLog.companyId, companyId)))
.toHaveLength(firstActivity.length);
const questionsQueue = queues.find((queue) => queue.key === "questions")!;
await db.update(decisionQueues).set({ seedRulesEnabled: false }).where(eq(decisionQueues.id, questionsQueue.id));
await decisionQueueService(db).materializeSeededQueues(companyId, [attentionItem({
sourceKind: "issue_thread_interaction",
sourceId: randomUUID(),
subjectKind: "interaction",
metadata: { kind: "ask_user_questions", issueId },
issueId,
})]);
expect(await db.select().from(decisionQueueItems).where(eq(decisionQueueItems.queueId, questionsQueue.id)))
.toHaveLength(1);
});
it("records agent decide-by, preserves override history, and exposes the board override attribution", async () => {
const { companyId, agentId, issueId } = await seed();
const agent = agentActor(companyId, agentId);
const board = boardActor(companyId, "override-user");
const agentSet = await request(app(agent))
.put(`/api/companies/${companyId}/decision-triage/review/${issueId}`)
.send({ decideBy: "today", snoozedUntil: "2026-08-03T12:00:00.000Z" })
.expect(200);
expect(agentSet.body).toMatchObject({
decideBy: "today",
setByType: "agent",
setByAgentId: agentId,
version: 1,
});
const overridden = await request(app(board))
.put(`/api/companies/${companyId}/decision-triage/review/${issueId}`)
.send({ decideBy: "2026-08-08", snoozedUntil: null })
.expect(200);
expect(overridden.body).toMatchObject({
decideBy: "2026-08-08",
setByType: "user",
setByUserId: "override-user",
version: 2,
});
const events = await db.select().from(decisionTriageEvents).where(and(
eq(decisionTriageEvents.companyId, companyId),
eq(decisionTriageEvents.sourceKind, "review"),
eq(decisionTriageEvents.sourceId, issueId),
));
expect(events).toHaveLength(2);
expect(events[0]?.actorAgentId).toBe(agentId);
expect(events[1]?.actorUserId).toBe("override-user");
expect(events[1]?.details).toMatchObject({ previousDecideBy: "today", decideBy: "2026-08-08" });
const current = await db.select().from(decisionTriage).where(eq(decisionTriage.sourceId, issueId));
expect(current[0]?.setByUserId).toBe("override-user");
});
it("serializes concurrent partial triage updates without losing state or reusing a version", async () => {
const { companyId, issueId } = await seed();
const board = boardActor(companyId);
const [decideByResult, snoozeResult] = await Promise.all([
request(app(board))
.put(`/api/companies/${companyId}/decision-triage/review/${issueId}`)
.send({ decideBy: "today" }),
request(app(board))
.put(`/api/companies/${companyId}/decision-triage/review/${issueId}`)
.send({ snoozedUntil: "2026-08-03T12:00:00.000Z" }),
]);
expect(decideByResult.status).toBe(200);
expect(snoozeResult.status).toBe(200);
expect([decideByResult.body.version, snoozeResult.body.version].sort()).toEqual([1, 2]);
const current = await request(app(board))
.get(`/api/companies/${companyId}/decision-triage/review/${issueId}`)
.expect(200);
expect(current.body).toMatchObject({
decideBy: "today",
snoozedUntil: "2026-08-03T12:00:00.000Z",
version: 2,
});
const events = await db.select().from(decisionTriageEvents).where(and(
eq(decisionTriageEvents.companyId, companyId),
eq(decisionTriageEvents.sourceKind, "review"),
eq(decisionTriageEvents.sourceId, issueId),
));
expect(events.map((event) => event.details.version).sort()).toEqual([1, 2]);
});
it("rejects task-bridge JWTs and returns the same 404 for missing and unauthorized sources", async () => {
const { companyId, agentId, approvalId } = await seed();
const bridge = {
...agentActor(companyId, agentId),
source: "agent_jwt",
keyId: randomUUID(),
keyScope: { kind: "task_bridge", parentIssueId: randomUUID() },
};
await request(app(bridge)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "blocked",
title: "Blocked",
}).expect(403);
const skillTest = {
...agentActor(companyId, agentId),
source: "agent_jwt",
keyScope: { kind: "skill_test", issueId: randomUUID() },
};
await request(app(skillTest)).post(`/api/companies/${companyId}/decision-queues`).send({
key: "skill-test-blocked",
title: "Skill test blocked",
}).expect(403);
await request(app(boardActor(companyId))).post(`/api/companies/${companyId}/decision-queues`).send({
key: "secure",
title: "Secure",
}).expect(201);
const standardAgent = app(agentActor(companyId, agentId));
const unauthorized = await request(standardAgent)
.post(`/api/companies/${companyId}/decision-queues/secure/items`)
.send({ sourceKind: "approval", sourceId: approvalId })
.expect(404);
const missing = await request(standardAgent)
.post(`/api/companies/${companyId}/decision-queues/secure/items`)
.send({ sourceKind: "approval", sourceId: randomUUID() })
.expect(404);
expect(unauthorized.body).toEqual(missing.body);
});
});