perf(heartbeat): skip unused session and holder reads for queued input (#22282)

## Thinking Path

> - Paperclip saves comments on a task while an agent is busy.
> - A second comment can join an existing queued run.
> - Only a new run records the starting session, yet admission read it
before choosing a run.
> - Admission can also bind an existing run and then use the previous
holder's agent name.
> - Read the session at insertion. Use the selected holder's key to
decide whether the existing run can accept the comment.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Admitting board assignments and saved task comments.

**Subsystem affected**

Server task admission.

**Current behavior**

Adding a comment to queued work reads a task session it does not use. A
replaced execution holder can leave its old key in this admission
decision.

**Proposed behavior**

Read the default session in the transaction that creates a run. Use the
selected run's execution-agent key when evaluating queued input.

**Additional context**

Builds on #22268, #22279 and #22280.

## What Changed

- Keep explicit resume lookup early because it determines task scope.
- Use the stored execution key if present. Scope the fallback agent
lookup to the company.
- When binding an existing run, use the same normalized key for the
issue write and the coalescing decision.

## Verification

All 280 tests passed across the PostgreSQL wake-queue adapter, heartbeat
queue, heartbeat run-state and responsible-user suites at a4d75b2d3e.
Three explicit-continuation tests and server typecheck also passed. The
benchmark retained one run, two receipts and both input IDs. Warm median
statements for two comments fell from 76 to 74. Single assignment and
single comment stayed at 36 and 37. The missing and stale
execution-holder regressions cover both process and Runner.

Across the stack, a separate local PostgreSQL run compared f0caee6f95
and a4d75b2d3e in base/tip/tip/base order. Each revision had 34 warm
measurements per case. Median admission times fell from 41.8 to 25.1 ms
for assignment, 42.6 to 25.5 ms for one comment, and 70.7 to 49.7 ms for
two comments. The agent slot was occupied. Counts exclude HTTP task and
comment creation and run execution.

## Risks

The default session is read later, inside the admission transaction. A
fresh run can now observe session changes committed before that read.
Explicit resume and reconstruction keep their identity rules. When
replacing a stale holder, a matching queued run now receives the comment
instead of deferring it under the previous holder's key.

## Model Used

OpenAI GPT-6.2 Sol (gpt-6.2-sol) implemented the change. OpenAI Codex
reviewed and qualified it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Nicky Leach authored and GitHub committed 2026-11-18 20:05:16 -08:00
1 parent 3dc6ccd8d8
commit 90d985b8b9
6 files changed
+70 -24

No files matched your search

@@ -29,6 +29,7 @@ import {
import {
createPostgresIssueWakeAdmission,
createPostgresWakeQueueAdapter,
createWakeAdmissionReader,
createWakeAdmissionWriter,
} from "./postgres.js";
import type { WakeQueuePostgresAdapterDeps } from "./postgres.js";
@@ -184,6 +185,26 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
return id;
}
it("uses the issue execution name key before reading the agent's current name", async () => {
const companyId = await seedCompany();
const agentId = await seedAgent({ companyId, name: "Renamed agent" });
await db.transaction(async (tx) => {
const select = vi.spyOn(tx, "select");
const reader = createWakeAdmissionReader(tx, companyId);
const input = { companyId, activeExecutionRunAgentId: agentId,
issueExecutionAgentNameKey: " Original Name ", agentNameKey: "original name" };
await expect(reader.isSameExecutionAgent({ ...input, companyId: randomUUID() }))
.rejects.toThrow("admission is locked for a different company");
await expect(reader.isSameExecutionAgent(input)).resolves.toBe(true);
await expect(reader.isSameExecutionAgent({ ...input, agentNameKey: "renamed agent" })).resolves.toBe(false);
expect(select).not.toHaveBeenCalled();
await expect(reader.isSameExecutionAgent({ ...input, issueExecutionAgentNameKey: null, agentNameKey: "renamed agent" }))
.resolves.toBe(true);
expect(select).toHaveBeenCalledTimes(1);
});
});
it("checks execution holds only when the scoped saved receipt exists, inside the issue admission transaction", async () => {
const companyId = await seedCompany();
const agentId = await seedAgent({ companyId });
@@ -1021,6 +1021,8 @@ export function createWakeAdmissionReader(rawTx: IssueAdmissionDbTransaction, ad
},
) {
assertAdmissionCompany(admittedCompanyId, companyId);
const storedNameKey = normalizeAgentNameKey(issueExecutionAgentNameKey);
if (storedNameKey !== null) return storedNameKey === agentNameKey;
const executionAgent = await tx
.select({ name: agents.name })
.from(agents)
@@ -1031,12 +1033,8 @@ export function createWakeAdmissionReader(rawTx: IssueAdmissionDbTransaction, ad
),
)
.then((rows) => rows[0] ?? null);
const executionAgentNameKey =
normalizeAgentNameKey(issueExecutionAgentNameKey) ??
normalizeAgentNameKey(executionAgent?.name);
return (
Boolean(executionAgentNameKey) && executionAgentNameKey === agentNameKey
);
const executionAgentNameKey = normalizeAgentNameKey(executionAgent?.name);
return executionAgentNameKey !== null && executionAgentNameKey === agentNameKey;
},
async findExistingDeferredWake(
@@ -82,7 +82,7 @@ type FreshIssueRunInput = {
failedChatRetryReceiptId?: string;
ownsReconstructionContinuation: boolean;
contextSnapshot: Record<string, unknown>;
sessionIdBefore: string | null;
resolveSessionIdBefore(): Promise<string | null>;
continuationAttempt: number;
explicitContinuationRunId: string | null;
retryOfRunId: string | null;
@@ -161,7 +161,7 @@ export function createFreshIssueRunAdmission<Run extends { id: string }>(store:
wakeupRequestId: primary.id,
retryOfRunId: input.reconciledSourceRunId || input.retryOfRunId,
contextSnapshot: adopted.length ? withQueuedCommentIdsInRunContext(contextSnapshot, commentIds, adoptedWakeIds) : contextSnapshot,
sessionIdBefore: input.explicitContinuationRunId ? null : input.sessionIdBefore,
sessionIdBefore: input.explicitContinuationRunId ? null : await input.resolveSessionIdBefore(),
continuationAttempt: input.continuationAttempt,
...(input.reconciledRestoreRetryCount !== null ? {
scheduledRetryAttempt: input.reconciledRestoreRetryCount,
+27 -3
View File
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { contextCompactionService } from "../context-compaction.js";
import { claimNativeSessionResumptions } from "../native-runtime/native-finalization-reconciler.js";
import type { Db } from "@paperclipai/db";
import { authUsers, agents, agentWakeupRequests, budgetIncidents, budgetPolicies, caseHandoffs, cases, chatCompletionDeliveries, companies, companyMemberships, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, heartbeatRuns, issueComments, issueThreadInteractions, issues, nativeRunFinalizations } from "@paperclipai/db";
import { authUsers, agents, agentTaskSessions, agentWakeupRequests, budgetIncidents, budgetPolicies, caseHandoffs, cases, chatCompletionDeliveries, companies, companyMemberships, costEvents, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, environmentLeases, heartbeatRuns, issueComments, issueThreadInteractions, issues, nativeRunFinalizations } from "@paperclipai/db";
import { eq, inArray, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
@@ -599,6 +599,12 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
payload: withQueuedCommentIdsInWakePayload({ issueId: issue.id, mutation }, [commentId]),
}))).returning();
const admit = deps.wakeQueue.admitIssueWake.getMockImplementation()!;
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admit(target, async (tx, admission) => {
await tx.insert(agentTaskSessions).values({ companyId: company.id, agentId: agent.id,
adapterType: agent.adapterType, taskKey: issue.id, sessionDisplayId: "session-available-in-admission" });
return evaluate(tx, admission);
}, effects));
const run = await queue.enqueueWakeup(agent.id, {
source: "assignment", triggerDetail: "system", reason: "issue_assigned", requestedByActorType: "system",
payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id, interruptedRunId: interruptedRun.id },
@@ -606,6 +612,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
const commentIds = matchingIssue ? [earlier.id, later.id] : [later.id];
expect(run).toMatchObject({ status: "queued", responsibleUserId: user.id, invocationSource: "assignment",
sessionIdBefore: "session-available-in-admission",
contextSnapshot: { wakeCommentIds: commentIds, wakeCommentId: later.id,
queuedCommentSourceWakeIds: matchingIssue ? [handoff.id, local.id] : [local.id] } });
const receipts = new Map((await db.select().from(agentWakeupRequests)).map(wake => [wake.id, wake]));
@@ -923,7 +930,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].executionRunId).toBe(f.source.id);
});
it.each((["process", "paperclip_runner"] as const).flatMap(adapterType => (["coalesced", "claimed-primary", "legacy-envelope", "delete-old", "delete-new", "stopped", "running", "system-primary", "foreign-primary", "same-named-agent", "mixed-author"] as const).map(scenario => ({ adapterType, scenario }))))("protects a queued comment batch across $adapterType/$scenario", async ({ adapterType, scenario }) => {
it.each((["process", "paperclip_runner"] as const).flatMap(adapterType => (["coalesced", "missing-execution-key", "stale-execution-key", "claimed-primary", "legacy-envelope", "delete-old", "delete-new", "stopped", "running", "system-primary", "foreign-primary", "same-named-agent", "mixed-author"] as const).map(scenario => ({ adapterType, scenario }))))("protects a queued comment batch across $adapterType/$scenario", async ({ adapterType, scenario }) => {
const f = await fixture("queued");
let admissionAgent: typeof agents.$inferSelect = { ...f.agent, adapterType, adapterConfig: adapterType === "paperclip_runner" ? { provider: "codex" } : {} };
await db.update(agents).set({ adapterType, adapterConfig: admissionAgent.adapterConfig }).where(eq(agents.id, f.agent.id));
@@ -951,6 +958,13 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
const [scanned] = await db.update(heartbeatRuns).set({ responsibleUserId: userId,
contextSnapshot: withQueuedCommentIdsInRunContext({ issueId: f.issue.id, wakeReason: "issue_commented" }, [oldComment.id]),
}).where(eq(heartbeatRuns.id, f.run.id)).returning();
if (scenario === "missing-execution-key" || scenario === "stale-execution-key") {
const [staleRun] = scenario === "stale-execution-key" ? await db.insert(heartbeatRuns).values({
companyId: f.company.id, agentId: f.agent.id, status: "succeeded", contextSnapshot: { issueId: f.issue.id },
}).returning() : [];
await db.update(issues).set({ executionRunId: staleRun?.id ?? null,
executionAgentNameKey: staleRun ? "previous agent name" : null }).where(eq(issues.id, f.issue.id));
}
let foreignPrimary: typeof agentWakeupRequests.$inferSelect | undefined;
if (scenario === "foreign-primary") {
const [company] = await db.insert(companies).values({ name: "Foreign source", issuePrefix: "FOREIGN" }).returning();
@@ -960,6 +974,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
}
if (scenario === "delete-old") await db.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, oldComment.id));
const deps = callbacks(db);
const resolveSession = vi.spyOn(deps, "resolveSessionBeforeForWakeup");
deps.getAgent.mockResolvedValue(admissionAgent);
deps.canStartNewExternalBotWork.mockResolvedValue(false);
deps.instanceSettings.getExperimental.mockResolvedValue({ enableIsolatedWorkspaces: false, enableNativeRunner: true } as Awaited<ReturnType<HeartbeatQueueDependencies["instanceSettings"]["getExperimental"]>>);
@@ -971,7 +986,12 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
mergeCoalescedContextSnapshot, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake },
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
deps.wakeQueue.admitIssueWake.mockImplementation(admission.admitIssueWake);
const busyAdmission = vi.fn();
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admission.admitIssueWake(target,
(tx, stores) => evaluate(tx, { ...stores, admitWakeBehindIssueExecution(input) {
busyAdmission(input.activeExecutionRun.id, input.issueExecutionAgentNameKey);
return stores.admitWakeBehindIssueExecution(input);
} }), effects));
const queue = createHeartbeatQueue(db, deps);
if (scenario === "running") {
deps.canStartNewExternalBotWork.mockResolvedValue(true);
@@ -1000,6 +1020,10 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
return;
}
expect(coalesced).toMatchObject({ id: f.run.id, status: "queued", contextSnapshot: { wakeCommentIds: [oldComment.id, newComment.id] } });
expect(resolveSession).not.toHaveBeenCalled();
if (scenario === "missing-execution-key" || scenario === "stale-execution-key") {
expect(busyAdmission).toHaveBeenCalledExactlyOnceWith(f.run.id, f.agent.name.toLowerCase());
}
const accepted = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.company.id));
expect(accepted.find(row => row.id !== f.wake.id)).toMatchObject({ status: "coalesced", runId: f.run.id, finishedAt: expect.any(Date),
requestedByActorType: "user", requestedByActorId: incomingActorId });
+9 -9
View File
@@ -3270,9 +3270,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// Old or direct timer producers cannot create a doomed taskless Runner
// run. Explicit session restoration above can supply a real task scope.
if (genericTimerWake && !supportsUnscopedTimerHeartbeat(agent)) return null;
const sessionBefore =
const resolveSessionBefore = async (tx: Pick<Db, "select">) =>
explicitResumeSession?.sessionDisplayId ??
(await resolveSessionBeforeForWakeup(agent, effectiveTaskKey));
resolveSessionBeforeForWakeup(agent, effectiveTaskKey, tx);
let hasResolvablePriorSessionWorkspace: boolean | null = null;
const resolveHasResolvablePriorSessionWorkspace = async () => {
if (hasResolvablePriorSessionWorkspace !== null)
@@ -4306,6 +4306,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.where(eq(heartbeatRuns.id, issue.executionRunId))
.then((rows) => rows[0] ?? null)
: null;
let activeExecutionAgentNameKey = issue.executionAgentNameKey;
const manualExecutionHold = activeExecutionRun && await nativeManualAdmissionHoldsExecution(tx as unknown as Db,
{ companyId: issue.companyId, runId: activeExecutionRun.id, issueId: issue.id });
@@ -4435,16 +4436,15 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const legacyAgent = await tx
.select({ name: agents.name })
.from(agents)
.where(eq(agents.id, legacyRun.agentId))
.where(and(eq(agents.companyId, issue.companyId), eq(agents.id, legacyRun.agentId)))
.then((rows) => rows[0] ?? null);
activeExecutionAgentNameKey = normalizeAgentNameKey(legacyAgent?.name);
await notifyStatusCardIssueChange(tx, issue.companyId);
await tx
.update(issues)
.set({
executionRunId: legacyRun.id,
executionAgentNameKey: normalizeAgentNameKey(
legacyAgent?.name,
),
executionAgentNameKey: activeExecutionAgentNameKey,
executionLockedAt: new Date(),
updatedAt: new Date(),
})
@@ -4777,7 +4777,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
issueId: issue.id,
agentId,
agentNameKey,
issueExecutionAgentNameKey: issue.executionAgentNameKey,
issueExecutionAgentNameKey: activeExecutionAgentNameKey,
activeExecutionRun: {
id: activeExecutionRun.id,
agentId: activeExecutionRun.agentId,
@@ -5064,7 +5064,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
? durableRequest!.failedRunRetry!.failedRunId
: opts.failedRunId ?? automaticParentRunId,
contextSnapshot: enrichedContextSnapshot,
sessionIdBefore: sessionBefore,
resolveSessionIdBefore: () => resolveSessionBefore(tx),
continuationAttempt,
reconciledSourceRunId,
reconciledRestoreRetryCount,
@@ -5299,7 +5299,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
responsibleUserId: await resolveQueuedResponsibleUserId(),
wakeupRequestId: wakeupRequest.id,
contextSnapshot: enrichedContextSnapshot,
sessionIdBefore: sessionBefore,
sessionIdBefore: await resolveSessionBefore(tx),
continuationAttempt,
})
.returning()
+7 -4
View File
@@ -1235,8 +1235,8 @@ export function createHeartbeatRunState(db: Db) {
}).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0] ?? null);
}
async function getRuntimeState(agentId: string) {
return db
async function getRuntimeState(agentId: string, reader: Pick<Db, "select"> = db) {
return reader
.select()
.from(agentRuntimeState)
.where(eq(agentRuntimeState.agentId, agentId))
@@ -1273,8 +1273,9 @@ export function createHeartbeatRunState(db: Db) {
agentId: string,
adapterType: string,
taskKey: string,
reader: Pick<Db, "select"> = db,
) {
return db
return reader
.select()
.from(agentTaskSessions)
.where(
@@ -1507,6 +1508,7 @@ export function createHeartbeatRunState(db: Db) {
async function resolveSessionBeforeForWakeup(
agent: typeof agents.$inferSelect,
taskKey: string | null,
reader: Pick<Db, "select"> = db,
) {
if (taskKey) {
const codec = getAdapterSessionCodec(agent.adapterType);
@@ -1515,6 +1517,7 @@ export function createHeartbeatRunState(db: Db) {
agent.id,
agent.adapterType,
taskKey,
reader,
);
const parsedParams = normalizeSessionParams(
codec.deserialize(existingTaskSession?.sessionParamsJson ?? null),
@@ -1526,7 +1529,7 @@ export function createHeartbeatRunState(db: Db) {
);
}
const runtimeForRun = await getRuntimeState(agent.id);
const runtimeForRun = await getRuntimeState(agent.id, reader);
return runtimeForRun?.sessionId ?? null;
}