mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps give agents access to tools through saved profiles and approval rules. > - A save replaces the full set of action entries and access bindings. > - It also updates each Ask first rule and each retired rule in a separate query. > - Adding one agent can therefore cause hundreds of database round trips. > - This pull request keeps unchanged rows and batches the required changes. > - The benefit is faster saves with the same action permissions. ## Linked Issues or Issue Description **What happened?** An app permission save repeats database work for actions and agents that did not change. With 100 actions and 500 retired rules, one agent addition makes 636 queries. Of these, 599 update approval policies. Related boundary coverage: Refs #19973. This change keeps that issue's Off and agent-selection protections. **Expected behavior** Adding an agent applies the saved Allowed, Ask first, and Off choices. The save changes only the rows that need a change. Independent approval rules remain in force. **Steps to reproduce** 1. Create an app with 100 active actions and 99 Ask first rules. 2. Add 500 disabled app approval rules. 3. Save access for three agents. Then add a fourth agent. 4. Count database queries during the save. **Paperclip version or commit** Baseline: `51d77648bc62`. **Deployment mode** Built from source. The reproduction uses disposable local PostgreSQL. ## What Changed - Keep action entries and access bindings when their saved values match. - Insert new bindings in one query. Remove only bindings and entries that no longer apply. - Read managed approval rules for the selected app. Skip unchanged rules and rules that are already disabled. - Batch new or changed Ask first rules and disable stale enabled rules in one query. - Keep existing policy conditions, priority, author, and creation time when a rule changes. - Add real PostgreSQL regression tests and a repeatable benchmark. Document independent approval rules. ## Verification - Red: the original regression made 2,995 policy writes across five saves and failed the zero-write check. - Green: 636 focused service and runtime tests passed after the rebase. CI supplies the broad checks required by `pnpm verify` (exit 2). - Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-app-permission-save.ts --baseline-ref=51d77648bc62 --output=/tmp/app-permission-save.json`. - Local benchmark: two warmups and six samples in alternating order. Restore three agents outside each timed sample, then add the fourth agent. Median save time fell from 458.51 ms to 38.05 ms (12.0 times faster). Query count fell from 636 to 32. Unchanged policy writes fell from 599 to zero. - Compare all saved profile, entry, binding, and policy values after mapping fixture IDs and excluding generated row IDs and write times. Check Allowed and Ask first decisions for the added agent. Tests also cover Off, an unassigned agent, additive access, removal, independent policies, company boundaries, and re-enabled rules. - CI passes on `069e2c76252a19ec3c11a061de0506eccb0463d1`: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/43162600023). This includes typecheck, build, server tests, and all eight browser shards. - [Greptile](https://github.com/paperclipai/paperclip/pull/21033#issuecomment-6815608829) gives the final head 5/5. Its one finding is resolved. The separate Paperclip Review check is still queued. Its review task stopped after a Runner failure. - Hosted request timing and the specific reported approval prompt are not verified. No live permissions changed. ## Risks - The save now retains unchanged row IDs and timestamps. Consumers must use permission values to determine access. - Existing transaction locks and permission checks remain in place. The runtime concurrency tests exercise these paths. - The benchmark measures a local service with synthetic data. It does not establish hosted endpoint latency. - No migration or startup change is required. ## Model Used OpenAI Codex, GPT-6. Exact provider model ID is not exposed in this session. Used reasoning, tool calls, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
151 lines
10 KiB
TypeScript
151 lines
10 KiB
TypeScript
import assert from "node:assert/strict";
|
|
import { execFileSync } from "node:child_process";
|
|
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
|
import { createRequire } from "node:module";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { performance } from "node:perf_hooks";
|
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
import {
|
|
agents, companies, createDb, startEmbeddedPostgresTestDatabase,
|
|
toolApplications, toolCatalogEntries, toolConnections, toolPolicies,
|
|
} from "../packages/db/src/index.js";
|
|
import { toolAccessService } from "../server/src/services/tool-access.js";
|
|
import { toolAccessPolicyService } from "../server/src/services/tool-access-policy.js";
|
|
|
|
const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15);
|
|
assert.ok(baselineRef, "Supply an explicit pre-fix --baseline-ref=<SHA>");
|
|
const output = process.argv.find(arg => arg.startsWith("--output="))?.slice(9);
|
|
const repository = fileURLToPath(new URL("../", import.meta.url));
|
|
const git = (...args: string[]) => execFileSync("git", args, { cwd: repository, encoding: "utf8" }).trim();
|
|
const baselineRevision = git("rev-parse", "--verify", `${baselineRef}^{commit}`);
|
|
const servicePath = "server/src/services/tool-access.ts";
|
|
const policyPath = "server/src/services/tool-access/catalog/default-access.ts";
|
|
const runtimeChanges = git("diff", baselineRevision, "--name-only", "--", "server/src", "packages/db/src", "packages/shared/src")
|
|
.split("\n").filter(Boolean).filter(name => !name.includes("/__tests__/") && !/\.test\.[cm]?ts$/.test(name));
|
|
assert.deepEqual(runtimeChanges.filter(name => ![servicePath, policyPath].includes(name)), [],
|
|
"Isolate additional changed runtime dependencies before measuring");
|
|
assert.equal(git("diff", baselineRevision, "--", "pnpm-lock.yaml"), "", "Dependencies must match the pinned baseline");
|
|
const requireFromServer = createRequire(new URL("../server/package.json", import.meta.url));
|
|
const scratch = await mkdtemp(path.join(tmpdir(), "app-permission-baseline-"));
|
|
function baselineSource(sourcePath: string) {
|
|
const source = execFileSync("git", ["show", `${baselineRevision}:${sourcePath}`], { cwd: repository, encoding: "utf8" });
|
|
return source.replace(/from "([^"]+)"/g, (_match, specifier: string) => {
|
|
const resolved = sourcePath === servicePath && specifier === "./tool-access/catalog/default-access.js"
|
|
? path.join(scratch, "default-access.mts")
|
|
: specifier.startsWith(".")
|
|
? fileURLToPath(new URL(specifier, new URL(sourcePath, pathToFileURL(`${repository}/`))))
|
|
: requireFromServer.resolve(specifier).replace(/\.cjs$/, ".js");
|
|
return `from ${JSON.stringify(resolved)}`;
|
|
});
|
|
}
|
|
const keepAlive = setInterval(() => {}, 1000);
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
|
let db: ReturnType<typeof createDb> | undefined;
|
|
try {
|
|
await writeFile(path.join(scratch, "default-access.mts"), baselineSource(policyPath));
|
|
await writeFile(path.join(scratch, "tool-access.mts"), baselineSource(servicePath));
|
|
const baseline = await import(pathToFileURL(path.join(scratch, "tool-access.mts")).href) as { toolAccessService: typeof toolAccessService };
|
|
database = await startEmbeddedPostgresTestDatabase("app-permission-benchmark-");
|
|
const client = db = createDb(database.connectionString);
|
|
type Statement = { sql: string; params: unknown[] };
|
|
let captured: Statement[] | undefined;
|
|
client.$client.options.debug = (_connection, sql, params) => {
|
|
captured?.push({ sql, params: [...params] });
|
|
};
|
|
const fixtures = [];
|
|
for (const [label, factory] of [["baseline", baseline.toolAccessService], ["treatment", toolAccessService]] as const) {
|
|
const [company] = await client.insert(companies).values({ name: "Benchmark", issuePrefix: `B${randomUUID().slice(0, 6)}` }).returning();
|
|
const roster = await client.insert(agents).values(Array.from({ length: 4 }, (_, i) => ({ companyId: company.id, name: `Agent ${i}` }))).returning();
|
|
const [application] = await client.insert(toolApplications).values({ companyId: company.id, name: "App", type: "mcp_http" }).returning();
|
|
const [connection] = await client.insert(toolConnections).values({ companyId: company.id, applicationId: application.id,
|
|
uid: randomUUID(), name: "App connection", transport: "mcp_remote", status: "active", enabled: true }).returning();
|
|
const catalog = await client.insert(toolCatalogEntries).values(Array.from({ length: 100 }, (_, i) => ({
|
|
companyId: company.id, applicationId: application.id, connectionId: connection.id,
|
|
name: `action_${i}`, toolName: `action_${i}`, riskLevel: "write" as const, status: "active" as const,
|
|
versionHash: `v${i}`, schemaHash: `s${i}`,
|
|
}))).returning();
|
|
const service = factory(client);
|
|
const save = (agentIds: string[]) => service.finishGalleryAppConnection(company.id, connection.id, {
|
|
access: { agentIds }, enabledCatalogEntryIds: catalog.map(entry => entry.id),
|
|
askFirstCatalogEntryIds: catalog.slice(1).map(entry => entry.id),
|
|
});
|
|
const initial = await save(roster.slice(0, 3).map(agent => agent.id));
|
|
await client.insert(toolPolicies).values(Array.from({ length: 500 }, (_, i) => ({
|
|
companyId: company.id, name: `Retired ${i}`, policyType: "require_approval" as const, enabled: false,
|
|
config: { source: "app_gallery_finish", connectionId: connection.id, catalogEntryId: randomUUID() },
|
|
})));
|
|
const ids = new Map([[company.id, "company"], [connection.id, "connection"], [application.id, "application"], [initial.profile.id, "profile"],
|
|
...roster.map((agent, i) => [agent.id, `agent_${i}`] as [string, string]),
|
|
...catalog.map((entry, i) => [entry.id, `catalog_${i}`] as [string, string])]);
|
|
function normalize(value: unknown): unknown {
|
|
if (Array.isArray(value)) return value.map(normalize).sort((a, b) => JSON.stringify(a).localeCompare(JSON.stringify(b)));
|
|
if (typeof value === "string") return ids.get(value) ?? value.replaceAll(connection.id, "connection").replaceAll(connection.id.slice(0, 8), "connection");
|
|
if (value && typeof value === "object") return Object.fromEntries(Object.entries(value)
|
|
.filter(([key]) => !["id", "createdAt", "updatedAt"].includes(key)).map(([key, nested]) => [key, normalize(nested)]));
|
|
return value;
|
|
}
|
|
fixtures.push({ label, save: () => save(roster.map(agent => agent.id)),
|
|
reset: () => save(roster.slice(0, 3).map(agent => agent.id)), normalize,
|
|
company, roster, connection, catalog, samples: [] as number[], statements: [] as Statement[], result: initial });
|
|
}
|
|
for (let i = 0; i < 2; i++) for (const fixture of fixtures) {
|
|
await fixture.reset();
|
|
await fixture.save();
|
|
}
|
|
for (let i = 0; i < 6; i++) {
|
|
for (const fixture of i % 2 ? [...fixtures].reverse() : fixtures) {
|
|
await fixture.reset();
|
|
captured = [];
|
|
const started = performance.now();
|
|
fixture.result = await fixture.save();
|
|
fixture.samples.push(performance.now() - started);
|
|
fixture.statements = captured;
|
|
captured = undefined;
|
|
}
|
|
}
|
|
const snapshot = (fixture: typeof fixtures[number]) => fixture.normalize({
|
|
profile: fixture.result.profile, entries: fixture.result.profileEntries,
|
|
bindings: fixture.result.profileBindings, policies: fixture.result.policies,
|
|
});
|
|
assert.deepEqual(snapshot(fixtures[0]), snapshot(fixtures[1]), "Complete saved profile/entry/binding/policy values must match except generated IDs and write timestamps");
|
|
const policy = toolAccessPolicyService(client);
|
|
for (const fixture of fixtures) {
|
|
for (const [entry, expected] of [[fixture.catalog[0], "allow"], [fixture.catalog[1], "require_approval"]] as const) {
|
|
const result = await policy.decide({ companyId: fixture.company.id,
|
|
actor: { actorType: "agent", actorId: fixture.roster[3].id, agentId: fixture.roster[3].id },
|
|
request: { connectionId: fixture.connection.id, catalogEntryId: entry.id, toolName: entry.toolName, arguments: {} } });
|
|
assert.equal(result.decision, expected);
|
|
}
|
|
}
|
|
const measurements = fixtures.map(fixture => {
|
|
const sorted = [...fixture.samples].sort((a, b) => a - b);
|
|
const policyWrites = fixture.statements.filter(statement => /^(insert into|update|delete from) "tool_policies"/i.test(statement.sql));
|
|
return { label: fixture.label, samplesMs: fixture.samples, medianMs: (sorted[2] + sorted[3]) / 2,
|
|
minMs: sorted[0], maxMs: sorted.at(-1), queriesPerSave: fixture.statements.length, policyWritesPerSave: policyWrites.length };
|
|
});
|
|
assert.equal(measurements[1].policyWritesPerSave, 0, "Unchanged policies must not be written");
|
|
assert.ok(measurements[1].queriesPerSave < measurements[0].queriesPerSave / 5, "Remove at least 80% of round trips on this fixture");
|
|
const plans = [];
|
|
for (const fixture of fixtures) {
|
|
const select = fixture.statements.find(statement => /^select .* from "tool_policies" where /i.test(statement.sql));
|
|
if (select) plans.push({ label: fixture.label, sql: select.sql,
|
|
plan: await client.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${select.sql}`, select.params as never[]) });
|
|
}
|
|
const report = { baselineRevision, treatmentRevision: git("rev-parse", "HEAD"), treatmentDiff: git("diff", "--", servicePath, policyPath),
|
|
postgres: await client.$client`select version(), current_setting('shared_buffers') as shared_buffers`,
|
|
fixture: { actions: 100, askFirst: 99, disabledPolicies: 500, agentsBefore: 3, agentsAfter: 4, warmups: 2, samples: 6, order: "alternating", reset: "restore three-agent selection outside each timed sample" },
|
|
measurements, speedup: measurements[0].medianMs / measurements[1].medianMs,
|
|
correctness: "Complete profile, entries, bindings, and policies match after mapping fixture IDs and excluding generated row IDs and write timestamps. Allowed and Ask first decisions match for the added agent.",
|
|
scope: "Local assembled-service timings with disposable PostgreSQL. Hosted endpoint latency is unverified.",
|
|
plans, statements: fixtures.map(fixture => ({ label: fixture.label, statements: fixture.statements })) };
|
|
if (output) await writeFile(output, JSON.stringify(report, null, 2));
|
|
console.log(JSON.stringify({ ...report, treatmentDiff: undefined, statements: undefined, plans: undefined }, null, 2));
|
|
} finally {
|
|
clearInterval(keepAlive);
|
|
await db?.$client.end();
|
|
await database?.cleanup();
|
|
await rm(scratch, { recursive: true, force: true });
|
|
}
|