fix(connectors): keep result classification alive during run progress (#20763)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - App connectors screen returned content before they give it to an
agent.
> - The model screen checks run identity and reserves its budget in one
transaction.
> - That transaction took a repeatable-read snapshot before it locked
the run.
> - A run event could change the row first and cause all four attempts
to fail.
> - This pull request uses READ COMMITTED for this admission under the
existing locks.
> - The benefit is that connector results can pass screening while run
progress continues.

## Linked Issues or Issue Description

Refs #20385 and #20484. These changes fixed other run accounting paths.
They did not change classifier admission.

**What happened?**

A connector read can return the accounting contention error before the
model screen starts. A real gateway test reproduces SQLSTATE `40001` at
`prompt-injection-prevention.ts` when it reads `heartbeat_runs FOR
SHARE`. Each attempt commits a run event after the transaction starts
its snapshot.

**Expected behavior**

Return screened connector content and save its classifier charge while
run events continue. Refuse paid classification if the run stops, its
identity changes, or access or budget checks fail.

**Steps to reproduce**

Run this test with the production source from `9574065a28`:

`cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts -t 'returns a screened
connector result'`

The test fails after four `40001` errors. It passes with this change.

## What Changed

- Use READ COMMITTED only for prompt-injection admission. Keep company,
project, configuration, agent, run and secret share locks.
- Keep classifier settlement at repeatable read. Keep provider calls
outside transaction retries.
- Test screened results, stored charges and settled holds under run
progress. Test four concurrent reads and refusal after a run stops or
changes identity.
- Document the transaction boundary.

### Hypothesis and thesis for a fix

**Hypothesis:** the classifier admission snapshot becomes stale before
its run share lock. The gateway regression confirms this cause in
current source. The historical connector error alone does not identify
its SQL statement.

**Thesis for a fix:** read current run state under the existing locks at
READ COMMITTED. This removes stale snapshots caused by run events. The
change belongs to classifier admission because every screened connector
result uses this service.

Investigation source: [Braintrust
pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/d5369c32-4c0f-43a1-af94-a01fef0a697b).
Access is required to read the traces.

**Proof:** baseline `9574065a28` failed with `40001` after four
attempts. The same single-read regression passed after the change. The
full focused suites then passed, including four concurrent reads, stored
charges and refusal controls.

## Verification

- Real PostgreSQL gateway regression: baseline red in 28.07 seconds;
candidate green in 23.16 seconds.
- `cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts
src/__tests__/prompt-injection-prevention.test.ts`: 54 tests passed.
After sync with master `1ac8b6a711`, all 54 passed again in 34.46
seconds.
- The tests use the real gateway, content guards, classifier service and
database. The MCP server and paid model are simulated. This is not a
live deployment test.
- Existing controls cover company boundaries, budgets, credentials,
configuration changes, interruption and receipt recovery.
- Staging health reported `16024ffa2`. Its classifier and accounting
source matches the reproduced baseline and includes #20385 and #20484.
The revision at the earlier failure times remains unknown.
- `pnpm verify` selected 2,512 of 3,303 test files because its
dependency policy retained 2,510 files conservatively. The broad local
run was stopped after the selection became clear. It is not a pass. The
focused 54-test result above is the local proof; required CI supplies
broad coverage. An earlier selection was also invalidated by a
concurrent commit.
- Full preflight could not finish because this host has no Cargo. Pinned
dependencies and shared/plugin SDK builds completed. The focused tests
passed without Rust.
- [Greptile reviewed this head at
5/5](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790238737).
No inline thread is open. [Paperclip completed its assessment at 5/5
with no
findings](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790312753).
[Required CI passed for head
`643bbe51faa70dcb17f372b9e85e3cbbcb724b9d`](https://github.com/paperclipai/paperclip/actions/runs/42981473514),
including typecheck, build, 12 server test shards, nine serialized
suites, eight E2E shards and Runner checks. Snyk and Socket passed. The
full inventory contains 58 completed check runs; the neutral Paperclip
check is a completed 5/5 assessment, not an incomplete review.

## Risks

- READ COMMITTED uses a new snapshot for each statement. Existing
admission locks still protect the run status, identity, configuration
and secret checks. Budget checks and reservation writes still commit
together.
- Concurrent independent admissions retain existing budget behavior.
This change adds no retry of paid provider work.
- No schema, API or saved receipt format changes.
- Historical failing SQL is unavailable. The source regression proves
this failure path; it does not prove that every historical example has
the same cause.
- The change is unmerged and not deployed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment model ID and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-13 12:55:37 -06:00
1 parent 1e33415f67
commit 2b1d0fc825
3 files changed
+96 -3

No files matched your search

+8
View File
@@ -2516,6 +2516,14 @@ error in `cause`, and is logged with both. A run that fails this way keeps the
generic `adapter_failed` code, so existing retry paths apply, and records that
bounded message with the SQLSTATE as its failure cause.
Prompt-injection classifier admission also uses READ COMMITTED. Its company,
project, configuration, agent, run and secret share locks retain the admission
fences. Run status and execution identity are read under the run lock, so a
concurrent run-event append cannot invalidate an earlier snapshot and discard
a connector result before classification. The budget check, invocation and
reservation still commit together. Classifier settlement keeps its separate
repeatable-read transaction and receipt replay rules.
Upgrade all controllers/workers to this accounting contract and drain incompatible
pre-upgrade database transactions before enabling independent writers. Do not run
pre-upgrade synchronous counter/evaluation writers beside unlocked writers.
@@ -20,6 +20,7 @@ import { providerUsageReceipt } from "../services/provider-usage.js";
import { adapterExecutionControls, createAdapterExecutionControl, registerAdapterExecutionControl } from "../services/adapter-execution-control.js";
import type { PluginToolDispatcher } from "../services/plugin-tool-dispatcher.js";
import { promptInjectionCorpus } from "./fixtures/prompt-injection-corpus.js";
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
let db: ReturnType<typeof createDb>, database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>, home: string;
beforeAll(async () => {
home = await realpath(await mkdtemp(path.join(os.tmpdir(), "paperclip-guard-gateway-")));
@@ -106,7 +107,7 @@ async function createRemoteMcpToolFixture(db: ReturnType<typeof createDb>, compa
const benign = promptInjectionCorpus[0]!.content;
const safe = { verdict: { violation: false, findings: [] }, receipt: { ...providerUsageReceipt(), inputTokens: 100, outputTokens: 10, costStatus: "reported", costCents: "0.0020000", pricingProvenance: { source: "provider_reported" } } };
async function fixture() {
async function fixture(gatewayDb?: (run: typeof heartbeatRuns.$inferSelect) => typeof db) {
const f = await createRunFixture(db);
await db.insert(companyMemberships).values({ companyId: f.company.id, principalType: "user", principalId: "alice", membershipRole: "owner", status: "active" });
const binding = await aiConnectionService(db).save(f.company.id, "alice", { provider: "openrouter", method: "api_key", name: "Classifier", ownership: "shared", apiKey: "fixture-key", agentIds: [], allAgents: true }, "fixture-key");
@@ -121,7 +122,7 @@ async function fixture() {
? { protocolVersion: "2025-06-18", capabilities: { tools: {} }, serverInfo: { name: "fixture", version: "1" } }
: { content: [{ type: "text", text: benign }] } });
});
const gateway = createToolGatewayService(db, { promptInjectionProvider: classifier, remoteHttpRequest, toolActionSigningSecret: "signing-fixture" });
const gateway = createToolGatewayService(gatewayDb?.(f.run) ?? db, { promptInjectionProvider: classifier, remoteHttpRequest, toolActionSigningSecret: "signing-fixture" });
const session = await gateway.createSession({ companyId: f.company.id, agentId: f.agent.id, runId: f.run.id });
const tool = (await gateway.listToolsForSession(session.token)).find(t => t.connectionId === remote.connection.id)!;
return { ...f, remote, classifier, remoteHttpRequest, gateway, session, tool };
@@ -134,7 +135,88 @@ async function evidence(companyId: string, toolInvocationId: string) {
const events = await db.select().from(toolCallEvents).where(eq(toolCallEvents.invocationId, toolInvocationId));
expect(events).toContainEqual(expect.objectContaining({ eventType: "call_completed", metadata: expect.objectContaining({ promptInjection: expect.objectContaining({ detector: "model", invocationId: invocations[0].id }) }) }));
}
/** Commit a real competing write after each accounting snapshot starts.
* The admission must still read current authority when it obtains its locks. */
function interleaveAccountingSnapshot(advance: () => Promise<void>) {
return new Proxy(db, { get(target, key) {
if (key !== "transaction") return Reflect.get(target, key);
return (callback: Parameters<typeof db.transaction>[0], options: Parameters<typeof db.transaction>[1]) => {
let first = Boolean(options?.isolationLevel);
return target.transaction(tx => callback(new Proxy(tx, { get(transaction, property) {
if (property !== "select") return Reflect.get(transaction, property);
return (...fields: Parameters<typeof db.select>) => {
const builder = transaction.select(...fields), from = builder.from.bind(builder);
builder.from = ((...args: Parameters<typeof from>) => {
const query = from(...args);
if (args[0] === companies && first) {
first = false;
const execute = query.execute.bind(query);
query.execute = async (...executeArgs: Parameters<typeof execute>) => {
const rows = await execute(...executeArgs);
await advance();
return rows;
};
}
return query;
}) as typeof builder.from;
return builder;
};
} })), options);
};
} });
}
describe("model classification at gateway result boundaries", () => {
it("returns a screened connector result while run events advance the admission snapshot", async () => {
let interleavings = 0;
const f = await fixture(run => interleaveAccountingSnapshot(async () => {
await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id,
agentId: run.agentId, eventType: "lifecycle", message: "stream advanced during classifier admission" });
interleavings++;
}));
const result = await f.gateway.executeTool({ sessionToken: f.session.token, tool: f.tool.name, parameters: {} });
expect(JSON.stringify(result.result)).toContain("secret.txt");
expect(interleavings).toBeGreaterThan(0);
expect(f.classifier).toHaveBeenCalledOnce();
await evidence(f.company.id, result.invocationId);
const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.company.id));
expect(holds).toHaveLength(1);
expect(holds[0].state).toBe("settled");
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(1);
const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(current.nextEventSeq).toBeGreaterThan(f.run.nextEventSeq);
});
it.each(["stopped", "steered"] as const)("refuses classifier dispatch when the run is %s after the admission snapshot", async change => {
const f = await fixture(run => interleaveAccountingSnapshot(async () => {
await db.update(heartbeatRuns).set(change === "stopped" ? { status: "cancelled" } : { responsibleUserId: "alice" })
.where(eq(heartbeatRuns.id, run.id));
}));
await expect(f.gateway.executeTool({ sessionToken: f.session.token, tool: f.tool.name, parameters: {} })).rejects.toMatchObject({
status: 422, reasonCode: "prompt_injection_blocked",
});
expect(f.classifier).not.toHaveBeenCalled();
expect(await db.select().from(promptInjectionInvocations).where(eq(promptInjectionInvocations.companyId, f.company.id))).toHaveLength(0);
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(0);
});
it("screens concurrent connector reads and saves each charge once while run events advance", async () => {
const f = await fixture(run => interleaveAccountingSnapshot(async () => {
await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, agentId: run.agentId,
eventType: "lifecycle", message: "parallel connector progress" });
}));
const results = await Promise.all(Array.from({ length: 4 }, () => f.gateway.executeTool({
sessionToken: f.session.token, tool: f.tool.name, parameters: {},
})));
expect(f.classifier).toHaveBeenCalledTimes(4);
expect(new Set(results.map(result => result.invocationId)).size).toBe(4);
for (const result of results) {
expect(JSON.stringify(result.result)).toContain("secret.txt");
await evidence(f.company.id, result.invocationId);
}
const holds = await db.select().from(budgetReservations).where(eq(budgetReservations.companyId, f.company.id));
expect(holds).toHaveLength(4);
expect(holds.every(hold => hold.state === "settled")).toBe(true);
expect(await db.select().from(costEvents).where(eq(costEvents.companyId, f.company.id))).toHaveLength(4);
});
it.each([
["direct", false], ["test", false], ["approved", false],
["direct", true], ["test", true], ["approved", true],
@@ -195,6 +195,9 @@ export function promptInjectionPreventionService(db: Db, options: {
try { credential = await aiConnectionService(db).credential(selected.connection, 0, { responsibleUserId: selected.userId,
actorType: selected.actorType, actorId: selected.actorId, issueId: selected.issueId, heartbeatRunId: selected.runId }); }
catch (error) { const reason = unavailableReason(error); if (reason) return { status: "unavailable", reason }; throw error; }
// The shared admission fences and run lock protect the live authority
// checks below. A snapshot taken before the run lock would go stale on
// every streamed run-event append, losing an otherwise valid tool result.
const admission = await withOperationAccountingTransaction(db, context.companyId, async (tx, publications) => {
const [company] = await tx.select({ status: companies.status }).from(companies)
.where(eq(companies.id, context.companyId)).for("share");
@@ -244,7 +247,7 @@ export function promptInjectionPreventionService(db: Db, options: {
projectId: current.projectId, amountCents: unitsToCents(amount), providerStartedAt: invocation.startedAt });
await queueBudgetEvaluation(tx, { companyId: context.companyId, agentId: current.agentId, projectId: current.projectId });
return { invocation };
});
}, { isolationLevel: "read committed" });
if (admission.reason) return { status: "unavailable", reason: admission.reason };
const invocation = admission.invocation!;
let outcome: PromptInjectionProviderOutcome;