mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs its background recovery queues on one Node main thread. Each database statement and each byte returned costs main-thread time. The target is 1,000 concurrent agents per server. > - On the idle hosted staging instance, the stranded-task check is the top remaining cost. Each check read the full agent row (with its configuration blobs) and then every agent of the company, only to decide whether this one agent can run. Per check it also opened a repeatable-read budget transaction for the same company and agent. > - The org-chain evaluation only looks at the agent's own managers. Reading the whole company is work proportional to the size of the company, for an answer that depends on a few rows. > - The same check also read the external review wait of each in-review task one task at a time. > - This pull request reads only the agent's reporting chain (one recursive query up `reports_to`), gives the stranded check a lean agent read, and reads the review waits of a pass's tasks in one call. > - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87 percent fewer rows, and no per-task review-wait chain for tasks that wait, with the same answers. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The per-task reads of the stranded-task recovery check (`reconcileStrandedAssignedIssues`) and of every caller of `evaluateAgentInvokabilityFromDb` and `agentService.getById`. **Subsystem affected** Server: issue recovery and agent invokability. **Current behavior** `evaluateAgentInvokabilityFromDb` reads `id, company_id, name, reports_to, status` for every agent of the company on each call. `getById` does the same for `orgChainHealth`. The stranded check calls it once per task, after a full-row agent read with the configuration blobs. On staging at idle these are the `agents where company_id = $1` read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The check also calls `listExternalReviewWaits` for one task at a time (the `chat_message_links` / `chat_github_reviews` reads at about 3/s on staging). **Proposed behavior** Read only the agent's managers, with the same cycle, missing-manager, cross-company and terminated rules. Read nothing when the agent has no manager or when its own status already decides the answer. Read the stranded check's agent with only the fields it uses. Read the review waits of all tasks of a pass in one call, and use that answer only to skip a task that waits. **Reason and benefit** At idle, the average result size on staging is about 17 KB per statement. The stranded check is about 22 statements and 17 KB per task, and it runs about 3 times per second there. ## What Changed - `server/src/lib/agent-reporting-chain.ts` (new): `readAgentReportingChain` reads the agent's managers with one recursive CTE up `reports_to`, limited to the agent's company. `UNION` stops the walk at a cycle. A root agent costs no statement. - `server/src/services/agent-invokability.ts`: `evaluateAgentInvokabilityFromDb` evaluates against the chain, not the company list. It reads nothing when the agent's own status is not invokable, because the status then decides the answer and the chain is not part of it. Every caller benefits with no change: wake enqueue, queued-run claim, retries, run dispatch, wake-queue promotion, continuation delivery, task watchdogs, thread interactions, native completion feedback, native session resumption, native review participants, agent conversations and recovery. - `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth` from the chain, not from the company list. - `server/src/services/recovery/service.ts`, stranded check: - The agent read selects only `id, company_id, name, reports_to, status, lifecycle_state, lifecycle_holds`, not the full row. - `hasPersistedDurableWaitPath` takes the pass. When the task is in the pass, the first in-review task reads the external review waits of every task of the pass with one `listExternalReviewWaits` call. A task in that answer waits and is skipped. A task not in that answer is read again alone, exactly as before, so a missing answer never lets a check go further than a fresh read would. - A company-wide call without a pass gets a pass of its in-review candidates. - `server/src/services/scoped-deadline-work.ts`: an optional `beforeChecks(entries)` hook gets the entries of the batch before their checks run. - `server/src/services/recovery/pass-reads.ts` (new) and `server/src/services/recovery/work.ts`: the task queues build a pass from the batch in `beforeChecks` and give it to each check of that batch. - Tests: - `server/src/__tests__/agent-reporting-chain.test.ts` (new): property test, see Verification. - `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new): statements and bytes per check, per pass and per full worker drain. - `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new): review-wait answers and reads, see Verification. - `server/src/__tests__/monthly-spend-service.test.ts`: the select-sequence stub no longer serves a company agent list to `getById`. - `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a company with an org chart, review tasks with succeeded native runs, and optional paused, terminated and recovery-action tasks. - `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional statement meter, installed before the first connection. - `server/src/__tests__/agent-record-read-projections.test.ts`: `getById` now makes one typed `agents` select; the chain is a raw query. ### Why the shared review-wait read cannot start or escalate work that a fresh read would not - A pass is one coordinator batch: at most 25 checks, one after the other. Its read is never older than the start of the batch. Nothing is kept across batches. - Only the positive answer is used: "this task waits on an external review or reply", which leads to a skip. A skip only delays the task to its next check, and the end of a wait sends its own hint. - A negative answer is not used. The task is read again alone, so every decision that goes further (an enqueue or an escalation) uses a fresh read, as today. - A first version also kept the agent's invokability and the budget answer for the pass. Review found that a kept permissive answer can reach an escalation, which does not re-check those facts the way an enqueue does. That reuse is removed: those reads stay per task. No authorization, pause, approval or budget check is removed or reordered. ## Verification - Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL): - 60 generated org charts, 2 companies each, up to 45 agents. Shapes: deep chains (depth 44), trees, full cycles and mixed graphs. - Managers are missing (inserted with foreign keys off), in another company, terminated, paused or pending approval at random depths. Agent statuses include an unknown value. - For each agent, the test compares the old list-based result with the new result. `evaluateAgentInvokabilityFromDb` deep-equals `evaluateAgentInvokability(agent, companyRows)`. `getAgentWorkEligibility` on the chain deep-equals it on the company list (this is the `getById` path). The test also asserts that every block reason and every chain-health reason occurred. - The list-based evaluation has no depth cap (`reporting_chain_too_deep` is declared but never returned), so the chain read has none either. A cycle ends the walk in both. - Mutation check: when the recursive step does not walk up, both tests fail. - Review-wait equality (`issue-recovery-review-wait-pass.test.ts`, embedded PostgreSQL): two tasks with a direct chat reply wait, one task with an inherited GitHub review wait, one task with none, and one task with an inherited GitHub review wait in another company. One call for all of them gives the same per-task answer as one call per task, for both companies. - Review-wait reads per pass (same file): the pass makes 2 bound-run message reads (one shared, one for the task whose candidate is not a wait) where per-task reads make 3, and 89 statements instead of 91. The check results are identical. On master the pass is ignored, so the test fails (3 is not less than 3). - Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents, 200 review tasks with succeeded native runs and no review waits, embedded PostgreSQL, statement meter on the client sockets): | | master | this branch | | --- | --- | --- | | One check: statements | 22 | 22 | | One check: bytes received | 17,170 | 8,617 | | One check: rows | 57 | 8 | | 25 checks: bytes | 429,331 | 215,505 | | Real worker, full company drain (200 checks): statements | 4,414 | 4,422 | | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 | | Real worker, full company drain: rows | 11,600 | 1,800 | | Company agent-list reads per drain | 200 | 0 | Here no task waits, so each batch pays one extra review-wait statement (8 per drain). Where tasks wait, the shared read replaces the per-task review, publication and authorization reads. - Regression after the review change, embedded PostgreSQL (664 tests): scoped-deadline-work, heartbeat-process-recovery, issue-recovery-deadlines, assigned-work-invariant, recovery-deadline-acceptance, heartbeat-issue-liveness-escalation, slack-conversation-lifecycle. Also issue-list-review-wait-authorization, external-review-wait, monthly-spend-service. - Regression of the first version, embedded PostgreSQL (757 tests): issue-assignee-invokability-routes, agent-record-read-projections, issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task, heartbeat-process-recovery, heartbeat-issue-liveness-escalation, assigned-work-invariant, recovery-deadline-acceptance, issue-recovery-actions, heartbeat-refused-assignment-wake, failed-continuation-continues-again, agent-invokability, agent-cross-tenant-authz-routes, heartbeat-stop-queued-task, issue-tree-pause-resume-run. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - A test double that gives `evaluateAgentInvokabilityFromDb` an object with only `select` now needs `execute`, and a select-sequence stub of `getById` gets one select fewer. CI found one (`monthly-spend-service`), fixed here. - A batch with no waiting task pays one extra cheap review-wait statement. - This branch and #21729 both touch `server/src/services/recovery/work.ts` in nearby lines. The second one to merge may need a small rebase. - `server/src/services/recovery/service.ts` was already past 1,000 lines. This change adds about 25 lines to it, beside the check it changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
155 lines
4.7 KiB
TypeScript
155 lines
4.7 KiB
TypeScript
import type { Db } from "@paperclipai/db";
|
|
import { agents } from "@paperclipai/db";
|
|
import { getAgentWorkEligibility, isAgentStatusInvokable, type AgentEligibilityAgent, type AgentOrgChainHealth } from "@paperclipai/shared";
|
|
import { readAgentReportingChain } from "../lib/agent-reporting-chain.js";
|
|
|
|
type AgentStatus = (typeof agents.$inferSelect)["status"];
|
|
|
|
export type AgentOrgRow = Pick<
|
|
typeof agents.$inferSelect,
|
|
"id" | "companyId" | "name" | "reportsTo" | "status"
|
|
>;
|
|
|
|
export type AgentInvokabilityBlockReason =
|
|
| "missing"
|
|
| "paused"
|
|
| "terminated"
|
|
| "pending_approval"
|
|
| "unknown_status"
|
|
| "manager_missing"
|
|
| "manager_company_mismatch"
|
|
| "manager_terminated"
|
|
| "reporting_cycle"
|
|
| "reporting_chain_too_deep";
|
|
|
|
export type AgentInvokability =
|
|
| { invokable: true }
|
|
| {
|
|
invokable: false;
|
|
reason: AgentInvokabilityBlockReason;
|
|
message: string;
|
|
details: Record<string, unknown>;
|
|
invalidOrgChain: boolean;
|
|
};
|
|
|
|
export const DIRECT_NON_INVOKABLE_STATUSES = new Set<AgentStatus>([
|
|
"paused",
|
|
"terminated",
|
|
"pending_approval",
|
|
]);
|
|
|
|
function blocked(
|
|
reason: AgentInvokabilityBlockReason,
|
|
message: string,
|
|
details: Record<string, unknown>,
|
|
invalidOrgChain = false,
|
|
): AgentInvokability {
|
|
return { invokable: false, reason, message, details, invalidOrgChain };
|
|
}
|
|
|
|
function statusBlockReason(status: AgentStatus): AgentInvokabilityBlockReason | null {
|
|
if (status === "paused") return "paused";
|
|
if (status === "terminated") return "terminated";
|
|
if (status === "pending_approval") return "pending_approval";
|
|
return null;
|
|
}
|
|
|
|
function toEligibilityAgent(row: AgentOrgRow): AgentEligibilityAgent {
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
name: row.name,
|
|
status: row.status,
|
|
reportsTo: row.reportsTo,
|
|
};
|
|
}
|
|
|
|
function invalidChainReason(health: AgentOrgChainHealth): AgentInvokabilityBlockReason {
|
|
if (health.reason === "terminated_ancestor") return "manager_terminated";
|
|
if (health.reason === "cycle") return "reporting_cycle";
|
|
return "manager_missing";
|
|
}
|
|
|
|
export function evaluateAgentInvokability(
|
|
agent: AgentOrgRow | null | undefined,
|
|
companyAgents: AgentOrgRow[],
|
|
): AgentInvokability {
|
|
if (!agent) {
|
|
return blocked("missing", "Agent no longer exists", {}, false);
|
|
}
|
|
|
|
const eligibility = getAgentWorkEligibility({
|
|
agent: toEligibilityAgent(agent),
|
|
agents: companyAgents.map(toEligibilityAgent),
|
|
});
|
|
|
|
if (eligibility.invokable) return { invokable: true };
|
|
|
|
const directStatusReason = eligibility.invokabilityReason === "unknown_status"
|
|
? "unknown_status"
|
|
: statusBlockReason(agent.status);
|
|
if (directStatusReason) {
|
|
return blocked(
|
|
directStatusReason,
|
|
"Agent is not invokable in its current state",
|
|
{ agentId: agent.id, agentStatus: agent.status },
|
|
false,
|
|
);
|
|
}
|
|
|
|
const health = eligibility.orgChainHealth;
|
|
const firstInvalidAncestor = health.firstInvalidAncestor;
|
|
return blocked(
|
|
invalidChainReason(health),
|
|
"Agent is not invokable because its reporting chain is invalid",
|
|
{
|
|
agentId: agent.id,
|
|
managerId: firstInvalidAncestor?.id ?? null,
|
|
managerStatus: firstInvalidAncestor?.status ?? null,
|
|
reportingChainAgentIds: health.fullChain
|
|
.filter((entry) => entry.relation === "ancestor")
|
|
.map((entry) => entry.id),
|
|
orgChainHealth: health,
|
|
},
|
|
true,
|
|
);
|
|
}
|
|
|
|
export async function evaluateAgentInvokabilityFromDb(
|
|
db: Db,
|
|
agent: AgentOrgRow | null | undefined,
|
|
): Promise<AgentInvokability> {
|
|
if (!agent || !isAgentStatusInvokable(agent.status)) return evaluateAgentInvokability(agent, []);
|
|
return evaluateAgentInvokability(agent, await readAgentReportingChain(db, agent));
|
|
}
|
|
|
|
export function listInvalidOrgChainDescendantIds(
|
|
terminatedAgentId: string,
|
|
companyAgents: AgentOrgRow[],
|
|
): string[] {
|
|
const byManager = new Map<string | null, AgentOrgRow[]>();
|
|
for (const row of companyAgents) {
|
|
const siblings = byManager.get(row.reportsTo ?? null) ?? [];
|
|
siblings.push(row);
|
|
byManager.set(row.reportsTo ?? null, siblings);
|
|
}
|
|
|
|
const invalidDescendantIds: string[] = [];
|
|
const stack = [...(byManager.get(terminatedAgentId) ?? [])];
|
|
const seen = new Set<string>([terminatedAgentId]);
|
|
while (stack.length > 0) {
|
|
const current = stack.pop();
|
|
if (!current || seen.has(current.id)) continue;
|
|
seen.add(current.id);
|
|
if (current.status !== "terminated") {
|
|
invalidDescendantIds.push(current.id);
|
|
}
|
|
stack.push(...(byManager.get(current.id) ?? []));
|
|
}
|
|
return invalidDescendantIds;
|
|
}
|
|
|
|
export function shouldCancelRunsForNonInvokableAgent(result: AgentInvokability) {
|
|
return !result.invokable && (result.reason === "terminated" || result.invalidOrgChain);
|
|
}
|