mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
perf(server): reuse sandbox provider schemas for five seconds on the remote-run path (#21758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs can use remote sandboxes that a sandbox provider plugin supplies (for example Daytona) > - Each sandbox operation resolves the environment config, and that step reads the provider config schema to find the secret-ref fields > - On a hosted instance with 7 remote runs, this schema read was the most frequent statement: 17.95 per second, about 2.5 per second for each run > - The server runs all agents on one Node main thread, so each statement costs main-thread time; at 1,000 agents this read alone is about 2,500 statements per second > - This pull request reuses the retained schema for a short, bounded time and drops it at once when the plugin registry changes a plugin manifest > - The benefit is that a live remote run reads the schema about once per 5 seconds for each provider, not once per operation ## Linked Issues or Issue Description **What existing behavior does this improve?** Every plugin sandbox operation (exec, file sync in/out, runner ingress attach, capability snapshot, workspace realize, lease resume, lease release) calls `resolveEnvironmentDriverConfigForRuntime`. That call runs the `with selected_provider as materialized ...` statement in `resolvePluginSandboxProviderConfigSchema`, every time. **Subsystem affected** Server: plugin environment driver (`server/src/services/plugin-environment-driver.ts`) and the plugin registry manifest writes. **Current behavior** The schema read runs once per operation. A census test that drives the real runtime calls of one remote run counts 19 schema reads for 19 operations (see the table below). The hosted measurement was 17.95 executions per second with 7 runs. **Proposed behavior** The read runs at most once per 5 seconds for each provider and database client. A plugin install, upgrade or uninstall through the plugin registry in the same process takes effect on the next operation. **Reason and benefit** The schema is instance-wide plugin manifest data. It does not change during normal runs. Re-reading it per operation spends main-thread and database time for no new information. **Breaking changes** None. No API, schema or saved-state change. ## What Changed - `resolvePluginSandboxProviderConfigSchema` now returns the retained schema without a database read when the entry was checked less than `PROVIDER_SCHEMA_REUSE_MS` (5,000 ms) ago and the plugin manifest revision did not change. After that, it uses the existing cheap revision check, as before. - A lookup sequence number stops an older, slower lookup from replacing a newer retained entry. - A missing provider is still never retained, so a provider that becomes available is used on the next operation. - New module `plugin-manifest-revision.ts` holds an in-process manifest revision counter. It is a separate module so that the registry and the driver do not import each other. - `plugin-registry.ts`: `install` (both the new-row and the reinstall paths), `update` and `uninstall` announce a manifest change after the write completes. This is 5 changed lines plus 1 import line. Status-only writes do not announce, because the schema read does not use plugin status. - `plugin-loader.ts`: `installPlugin` runs `registry.install` inside an outer transaction. It now announces the manifest change again after that transaction settles, so a schema read made before the commit cannot be reused after it. This is 1 changed line plus 1 import line. - Statement census per operation, before and after. One remote run, one secret-ref field in the provider config, embedded test database, fake in-process plugin worker: | Operation | Count | Schema reads before | Schema reads after | All statements before | All statements after | | --- | --- | --- | --- | --- | --- | | Lease acquire | 1 | 1 | 1 | 26 | 26 | | Runner attach (ingress endpoint) | 1 | 2 | 0 | 11 | 9 | | Capability snapshot | 1 | 1 | 0 | 6 | 5 | | Exec | 10 | 10 | 0 | 50 | 40 | | File write (sync in) | 1 | 1 | 0 | 5 | 4 | | File read (sync out) | 1 | 1 | 0 | 5 | 4 | | Workspace realize | 1 | 1 | 0 | 5 | 4 | | Lease resume (reattach) | 1 | 1 | 0 | 6 | 5 | | Lease release | 1 | 1 | 0 | 18 | 17 | | **Total** | 18 | **19** | **1** | 132 | 114 | - Other reads that stay on each operation (reported, not changed in this pull request): each exec resolves each secret-ref field with 3 reads (secret, binding, version) and 1 `secret_access_events` insert. Runner attach and the capability snapshot also read the plugin row by id. These are part of the secret authorization and audit path, so they keep their own checks. ## Verification - `cd server && npx vitest run src/__tests__/plugin-sandbox-provider-schema-census.test.ts` (new). On master the census test fails with 19 schema reads. With this change it passes with 1. - The new tests also prove these points: - A registry upgrade that changes the secret-ref fields takes effect on the next call in the same process. - A direct manifest change (as from another server process) takes effect after the 5-second bound, and no read occurs inside the bound. - A registry uninstall takes effect at once. - A plugin reinstall through `pluginLoader.installPlugin` takes effect after its install transaction commits, even when a schema read ran inside the uncommitted transaction. This test failed before the post-commit announcement and passes with it. - A missing provider is not retained. The next call after install reads and uses it. - Two companies share the provider schema but each gets only its own secret values. A cross-company secret id still fails with "Secret must belong to same company". - A new database client (a process restart) starts cold, as before. - `cd server && npx vitest run src/__tests__/plugin-database.test.ts`: passed (plugin install transaction paths). - `cd server && npx vitest run src/__tests__/sandbox-provider-schema-reads.test.ts`: the existing tests now step the clock past the bound where they change the database directly. - `cd server && npx vitest run src/__tests__/environment-runtime.test.ts`: 210 passed. - `cd server && npx vitest run src/__tests__/bundled-plugin-registration.test.ts src/__tests__/external-object-refresh-work.test.ts src/__tests__/plugin-lifecycle-restart.test.ts src/__tests__/plugin-loader-error-retry.test.ts src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts src/__tests__/plugin-sandbox-provider-config-validation.test.ts src/__tests__/environment-probe.test.ts`: 82 passed, 1 skipped. - `cd server && pnpm typecheck`: passed. ## Risks - Staleness bound: a plugin manifest change made by a different server process (for example during an overlapping deploy) can take up to 5 seconds to apply here. During that time, a secret-ref field that an upgrade adds is not resolved yet, and a field that an upgrade removes is still resolved. The bound is short and it applies only to which config paths are secret refs. - The retained value is instance-wide plugin manifest data, keyed by database client and driver key. It contains no company data and no secret values. It does not take part in an authorization decision. Secret resolution, the environment binding check and the access audit still run on each operation for the calling company. - Memory stays bounded by the existing limits: 32 entries per database client, 64 KiB per schema. - `plugin-registry.ts` has a small edit in `install`, `update` and `uninstall`. Other open work in that file can cause a simple merge conflict. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #17581 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
15afc32137
commit
56654adecd
7 files changed
+372
-10
No files matched your search
@@ -0,0 +1,313 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
agents,
|
||||
companies,
|
||||
createDb,
|
||||
environments,
|
||||
heartbeatRuns,
|
||||
plugins,
|
||||
type Db,
|
||||
} from "@paperclipai/db";
|
||||
import type { Environment, PaperclipPluginManifestV1, PluginEnvironmentDriverDeclaration } from "@paperclipai/shared";
|
||||
import { environmentRuntimeService } from "../services/environment-runtime.ts";
|
||||
import { resolveSandboxProviderSecretRefPaths } from "../services/environment-config.ts";
|
||||
import { pluginRegistryService } from "../services/plugin-registry.ts";
|
||||
import { pluginLoader } from "../services/plugin-loader.ts";
|
||||
import { secretService } from "../services/secrets.ts";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { meterDatabaseStatements } from "./helpers/database-statement-meter.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const PROVIDER_SCHEMA_STATEMENT = "with selected_provider as materialized";
|
||||
const SCHEMA_REUSE_BOUND_MS = 5_000;
|
||||
|
||||
describe.skipIf(!support.supported)("sandbox provider schema reads on the live remote-run path", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let seedDb: Db;
|
||||
const clients: Db[] = [];
|
||||
const meters: Array<ReturnType<typeof meterDatabaseStatements>> = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("sandbox-schema-census-");
|
||||
seedDb = createDb(database.connectionString);
|
||||
}, 120_000);
|
||||
afterEach(async () => {
|
||||
vi.useRealTimers();
|
||||
meters.splice(0).reverse().forEach(meter => meter.uninstall());
|
||||
await Promise.all(clients.splice(0).map(client => client.$client.end({ timeout: 1 })));
|
||||
});
|
||||
afterAll(async () => {
|
||||
await seedDb?.$client.end({ timeout: 1 });
|
||||
await database?.cleanup();
|
||||
});
|
||||
|
||||
function meteredClient() {
|
||||
const client = createDb(database.connectionString);
|
||||
clients.push(client);
|
||||
const meter = meterDatabaseStatements(client, database.connectionString);
|
||||
meters.push(meter);
|
||||
return { client, meter };
|
||||
}
|
||||
|
||||
function declaration(driverKey: string, secretPaths: string[]): PluginEnvironmentDriverDeclaration {
|
||||
return {
|
||||
driverKey, kind: "sandbox_provider", displayName: "Census provider",
|
||||
sandboxCapabilities: { runnerWebSocketIngress: true },
|
||||
configSchema: { type: "object", properties: Object.fromEntries([
|
||||
["template", { type: "string" }],
|
||||
...secretPaths.map(path => [path, { type: "string", format: "secret-ref" }]),
|
||||
]) },
|
||||
} as PluginEnvironmentDriverDeclaration;
|
||||
}
|
||||
|
||||
function manifest(pluginKey: string, declarations: PluginEnvironmentDriverDeclaration[], version = "1.0.0"): PaperclipPluginManifestV1 {
|
||||
return {
|
||||
id: pluginKey, apiVersion: 1, version, displayName: "Census provider", description: "Fixture",
|
||||
author: "Test", categories: ["automation"], capabilities: ["environment.drivers.register"],
|
||||
entrypoints: { worker: "worker.js" }, environmentDrivers: declarations,
|
||||
};
|
||||
}
|
||||
|
||||
async function seedPlugin(driverKey: string, secretPaths = ["apiKey"]) {
|
||||
const pluginKey = `test.census.${randomUUID()}`;
|
||||
const installed = await pluginRegistryService(seedDb).install(
|
||||
{ packageName: "@test/census" }, manifest(pluginKey, [declaration(driverKey, secretPaths)]));
|
||||
await seedDb.update(plugins).set({ status: "ready" }).where(eq(plugins.id, installed!.id));
|
||||
return { pluginId: installed!.id, pluginKey };
|
||||
}
|
||||
|
||||
async function seedCompanyEnvironment(driverKey: string, secretValue: string) {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), environmentId = randomUUID(), runId = randomUUID();
|
||||
await seedDb.insert(companies).values({ id: companyId, name: "Census", status: "active", issuePrefix: `C${companyId.slice(0, 6).toUpperCase()}` });
|
||||
await seedDb.insert(agents).values({
|
||||
id: agentId, companyId, name: "Runner", role: "engineer", status: "active",
|
||||
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {},
|
||||
});
|
||||
const secret = await secretService(seedDb).create(companyId, {
|
||||
name: `census-api-key-${randomUUID()}`, provider: "local_encrypted", value: secretValue,
|
||||
});
|
||||
const config = { provider: driverKey, template: "base", apiKey: secret.id, reuseLease: false };
|
||||
const now = new Date();
|
||||
await seedDb.insert(environments).values({
|
||||
id: environmentId, name: `census-${environmentId.slice(0, 8)}`, driver: "sandbox", status: "active", config,
|
||||
createdAt: now, updatedAt: now,
|
||||
});
|
||||
await secretService(seedDb).createBinding({
|
||||
companyId, secretId: secret.id, targetType: "environment", targetId: environmentId, configPath: "apiKey",
|
||||
});
|
||||
await seedDb.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "manual", status: "running" });
|
||||
const environment: Environment = {
|
||||
id: environmentId, name: `census-${environmentId.slice(0, 8)}`, description: null, driver: "sandbox",
|
||||
status: "active", config, envVars: {}, metadata: null, createdAt: now, updatedAt: now,
|
||||
};
|
||||
return { companyId, environment, runId, secretId: secret.id };
|
||||
}
|
||||
|
||||
function workerManager(pluginId: string, driverKey: string) {
|
||||
const configs: Array<{ method: string; config: Record<string, unknown> }> = [];
|
||||
const manager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
getWorker: vi.fn(() => ({ status: "running", supportedMethods: [
|
||||
"environmentAcquireLease", "environmentResumeLease", "environmentReleaseLease", "environmentDestroyLease",
|
||||
"environmentExecute", "environmentSyncIn", "environmentSyncOut", "environmentRealizeWorkspace",
|
||||
"environmentRunnerIngressEndpoint",
|
||||
] })),
|
||||
call: vi.fn(async (_id: string, method: string, params: { config: Record<string, unknown>; path?: string }) => {
|
||||
configs.push({ method, config: params.config });
|
||||
switch (method) {
|
||||
case "environmentAcquireLease":
|
||||
case "environmentResumeLease":
|
||||
return { providerLeaseId: "census-sandbox", metadata: { provider: driverKey, template: "base", remoteCwd: "/workspace" } };
|
||||
case "environmentExecute":
|
||||
return { exitCode: 0, signal: null, timedOut: false, stdout: "ok\n", stderr: "" };
|
||||
case "environmentSyncIn":
|
||||
case "environmentSyncOut":
|
||||
return { results: [] };
|
||||
case "environmentRealizeWorkspace":
|
||||
return { cwd: "/workspace", metadata: {} };
|
||||
case "environmentRunnerIngressEndpoint":
|
||||
return { kind: "authenticated_websocket", websocketUrl: `wss://runner.example.test${params.path}`,
|
||||
secretHeaders: [{ name: "x-runner-token", value: "token" }], generation: "g1" };
|
||||
case "environmentReleaseLease":
|
||||
return undefined;
|
||||
default:
|
||||
throw new Error(`Unexpected plugin method: ${method}`);
|
||||
}
|
||||
}),
|
||||
} as unknown as PluginWorkerManager;
|
||||
return { manager, configs };
|
||||
}
|
||||
|
||||
async function runCensus(client: Db, meter: ReturnType<typeof meterDatabaseStatements>, driverKey: string,
|
||||
seeded: Awaited<ReturnType<typeof seedCompanyEnvironment>>, pluginId: string) {
|
||||
const { manager, configs } = workerManager(pluginId, driverKey);
|
||||
const runtime = environmentRuntimeService(client, { pluginWorkerManager: manager });
|
||||
const rows: Array<{ operation: string; providerSchemaReads: number; statements: number }> = [];
|
||||
async function measure<T>(operation: string, run: () => Promise<T>) {
|
||||
meter.startLog();
|
||||
const value = await run();
|
||||
const log = meter.stopLog();
|
||||
rows.push({ operation, providerSchemaReads: log.filter(entry => entry.sql.includes(PROVIDER_SCHEMA_STATEMENT)).length,
|
||||
statements: log.length });
|
||||
return value;
|
||||
}
|
||||
const { environment } = seeded;
|
||||
const acquired = await measure("lease acquire", () => runtime.acquireRunLease({
|
||||
companyId: seeded.companyId, environment, issueId: null, heartbeatRunId: seeded.runId, persistedExecutionWorkspace: null,
|
||||
}));
|
||||
const lease = acquired.lease;
|
||||
await measure("runner attach (ingress endpoint)", () => runtime.getRunnerIngressEndpoint({ environment, lease, port: 8080, path: "/runner" }));
|
||||
await measure("capability snapshot", () => runtime.resolveCapabilities({ environment, lease }));
|
||||
await measure("exec", () => runtime.execute({ environment, lease, command: "true", args: [], cwd: "/workspace", env: {} }));
|
||||
for (let index = 0; index < 9; index++) {
|
||||
await measure("exec (repeat)", () => runtime.execute({ environment, lease, command: "true", args: [], cwd: "/workspace", env: {} }));
|
||||
}
|
||||
await measure("file write (sync in)", () => runtime.syncIn({ environment, lease, operations: [] }));
|
||||
await measure("file read (sync out)", () => runtime.syncOut({ environment, lease, operations: [] }));
|
||||
await measure("workspace realize", () => runtime.realizeWorkspace({ environment, lease, workspace: { remotePath: "/workspace" } }));
|
||||
await measure("lease resume (reattach)", () => runtime.resumeRunLease({ environment, lease }));
|
||||
await measure("lease release", () => runtime.releaseRunLeases(seeded.runId));
|
||||
const merged = new Map<string, { operation: string; count: number; providerSchemaReads: number; statements: number }>();
|
||||
for (const row of rows) {
|
||||
const entry = merged.get(row.operation) ?? { operation: row.operation, count: 0, providerSchemaReads: 0, statements: 0 };
|
||||
entry.count += 1;
|
||||
entry.providerSchemaReads += row.providerSchemaReads;
|
||||
entry.statements += row.statements;
|
||||
merged.set(row.operation, entry);
|
||||
}
|
||||
return { table: [...merged.values()], configs };
|
||||
}
|
||||
|
||||
it("reads the provider schema at most once per bound across a whole remote run, not once per operation", async () => {
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
const driverKey = `census-${randomUUID().slice(0, 8)}`;
|
||||
const { pluginId } = await seedPlugin(driverKey);
|
||||
const seeded = await seedCompanyEnvironment(driverKey, "company-a-key");
|
||||
const { client, meter } = meteredClient();
|
||||
const { table, configs } = await runCensus(client, meter, driverKey, seeded, pluginId);
|
||||
console.info(`provider schema census\n${table.map(row =>
|
||||
`${row.operation} x${row.count}: provider schema reads=${row.providerSchemaReads}, statements=${row.statements}`).join("\n")}`);
|
||||
const total = table.reduce((sum, row) => sum + row.providerSchemaReads, 0);
|
||||
const acquire = table.find(row => row.operation === "lease acquire")!;
|
||||
expect(acquire.providerSchemaReads).toBeLessThanOrEqual(1);
|
||||
expect(total).toBeLessThanOrEqual(1);
|
||||
expect(configs.filter(entry => entry.method !== "environmentAcquireLease").every(entry => entry.config.apiKey === "company-a-key")).toBe(true);
|
||||
});
|
||||
|
||||
it("applies a plugin upgrade that changes the schema at once in this process, and a change made elsewhere within the bound", async () => {
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
const driverKey = `census-${randomUUID().slice(0, 8)}`;
|
||||
const { pluginId, pluginKey } = await seedPlugin(driverKey, ["apiKey"]);
|
||||
const { client, meter } = meteredClient();
|
||||
expect(await resolveSandboxProviderSecretRefPaths(client, driverKey)).toEqual(new Set(["apiKey"]));
|
||||
await pluginRegistryService(seedDb).update(pluginId, {
|
||||
version: "2.0.0", manifest: manifest(pluginKey, [declaration(driverKey, ["apiKey", "token"])], "2.0.0"),
|
||||
});
|
||||
expect(await resolveSandboxProviderSecretRefPaths(client, driverKey)).toEqual(new Set(["apiKey", "token"]));
|
||||
|
||||
await seedDb.update(plugins).set({ manifestJson: manifest(pluginKey, [declaration(driverKey, ["token"])], "3.0.0") })
|
||||
.where(eq(plugins.id, pluginId));
|
||||
meter.startLog();
|
||||
const withinBound = await resolveSandboxProviderSecretRefPaths(client, driverKey);
|
||||
expect(meter.stopLog()).toHaveLength(0);
|
||||
expect([new Set(["apiKey", "token"]), new Set(["token"])]).toContainEqual(withinBound);
|
||||
vi.setSystemTime(Date.now() + SCHEMA_REUSE_BOUND_MS + 1);
|
||||
expect(await resolveSandboxProviderSecretRefPaths(client, driverKey)).toEqual(new Set(["token"]));
|
||||
|
||||
await pluginRegistryService(seedDb).uninstall(pluginId, true);
|
||||
expect(await resolveSandboxProviderSecretRefPaths(client, driverKey)).toEqual(new Set());
|
||||
});
|
||||
|
||||
it("never remembers a missing provider, so a provider that becomes ready is used on the next operation", async () => {
|
||||
const driverKey = `census-${randomUUID().slice(0, 8)}`;
|
||||
const { client, meter } = meteredClient();
|
||||
expect(await resolveSandboxProviderSecretRefPaths(client, driverKey)).toEqual(new Set());
|
||||
await seedDb.insert(plugins).values({
|
||||
pluginKey: `test.census.${randomUUID()}`, packageName: "@test/census", version: "1.0.0", installOrder: 0,
|
||||
status: "ready", manifestJson: manifest(`test.census.late`, [declaration(driverKey, ["apiKey"])]),
|
||||
});
|
||||
meter.startLog();
|
||||
expect(await resolveSandboxProviderSecretRefPaths(client, driverKey)).toEqual(new Set(["apiKey"]));
|
||||
expect(meter.stopLog().filter(entry => entry.sql.includes(PROVIDER_SCHEMA_STATEMENT))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps each company's secret resolution and binding check per operation while the provider schema is shared", async () => {
|
||||
const driverKey = `census-${randomUUID().slice(0, 8)}`;
|
||||
const { pluginId } = await seedPlugin(driverKey);
|
||||
const first = await seedCompanyEnvironment(driverKey, "company-a-key");
|
||||
const second = await seedCompanyEnvironment(driverKey, "company-b-key");
|
||||
const { client, meter } = meteredClient();
|
||||
const a = await runCensus(client, meter, driverKey, first, pluginId);
|
||||
const b = await runCensus(client, meter, driverKey, second, pluginId);
|
||||
expect(a.configs.filter(entry => entry.method !== "environmentAcquireLease").map(entry => entry.config.apiKey))
|
||||
.toEqual(expect.arrayContaining(["company-a-key"]));
|
||||
expect(a.configs.some(entry => entry.config.apiKey === "company-b-key")).toBe(false);
|
||||
expect(b.configs.filter(entry => entry.method !== "environmentAcquireLease").every(entry => entry.config.apiKey === "company-b-key")).toBe(true);
|
||||
|
||||
const crossCompany = { ...second.environment, config: { ...second.environment.config, apiKey: first.secretId } };
|
||||
const { manager } = workerManager(pluginId, driverKey);
|
||||
const runtime = environmentRuntimeService(client, { pluginWorkerManager: manager });
|
||||
await expect(runtime.execute({
|
||||
environment: crossCompany,
|
||||
lease: { companyId: second.companyId, environmentId: second.environment.id, issueId: null, providerLeaseId: "census-sandbox",
|
||||
metadata: { provider: driverKey, pluginId, sandboxProviderPlugin: true, template: "base", apiKey: first.secretId } } as never,
|
||||
command: "true", args: [], cwd: "/workspace", env: {},
|
||||
})).rejects.toThrow("Secret must belong to same company");
|
||||
});
|
||||
|
||||
it("applies a plugin reinstall only after its install transaction commits", async () => {
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
const driverKey = `census-${randomUUID().slice(0, 8)}`;
|
||||
const pluginKey = `test.census.reinstall${randomUUID().slice(0, 8)}`;
|
||||
await seedDb.insert(plugins).values({
|
||||
pluginKey, packageName: pluginKey, version: "1.0.0", installOrder: 0, status: "uninstalled",
|
||||
manifestJson: manifest(pluginKey, [declaration(driverKey, ["apiKey"])]),
|
||||
});
|
||||
const upgraded = manifest(pluginKey, [declaration(driverKey, ["apiKey", "token"])], "2.0.0");
|
||||
const packageRoot = await mkdtemp(path.join(os.tmpdir(), "census-plugin-"));
|
||||
try {
|
||||
await writeFile(path.join(packageRoot, "package.json"), JSON.stringify({
|
||||
name: pluginKey, version: upgraded.version, type: "module", paperclipPlugin: { manifest: "./manifest.js" },
|
||||
}));
|
||||
await writeFile(path.join(packageRoot, "manifest.js"), `export default ${JSON.stringify(upgraded)};\n`);
|
||||
await mkdir(path.join(packageRoot, "dist"), { recursive: true });
|
||||
await writeFile(path.join(packageRoot, "dist", "worker.js"), "export {};\n");
|
||||
const { client: observer } = meteredClient();
|
||||
expect(await resolveSandboxProviderSecretRefPaths(observer, driverKey)).toEqual(new Set(["apiKey"]));
|
||||
const installer = createDb(database.connectionString);
|
||||
clients.push(installer);
|
||||
const readBeforeCommit = new Proxy(installer, {
|
||||
get(target, property, receiver) {
|
||||
if (property !== "transaction") return Reflect.get(target, property, receiver);
|
||||
return (work: (tx: unknown) => Promise<unknown>) => target.transaction(async tx => {
|
||||
const result = await work(tx);
|
||||
expect(await resolveSandboxProviderSecretRefPaths(observer, driverKey)).toEqual(new Set(["apiKey"]));
|
||||
return result;
|
||||
});
|
||||
},
|
||||
});
|
||||
await pluginLoader(readBeforeCommit, { enableLocalFilesystem: false, enableNpmDiscovery: false })
|
||||
.installPlugin({ localPath: packageRoot });
|
||||
expect(await resolveSandboxProviderSecretRefPaths(observer, driverKey)).toEqual(new Set(["apiKey", "token"]));
|
||||
} finally {
|
||||
await rm(packageRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("starts cold on a new process, as before", async () => {
|
||||
const driverKey = `census-${randomUUID().slice(0, 8)}`;
|
||||
await seedPlugin(driverKey);
|
||||
const first = meteredClient();
|
||||
await resolveSandboxProviderSecretRefPaths(first.client, driverKey);
|
||||
const restarted = meteredClient();
|
||||
restarted.meter.startLog();
|
||||
expect(await resolveSandboxProviderSecretRefPaths(restarted.client, driverKey)).toEqual(new Set(["apiKey"]));
|
||||
expect(restarted.meter.stopLog().filter(entry => entry.sql.includes(PROVIDER_SCHEMA_STATEMENT))).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -21,6 +21,7 @@ import { readProcessStartedAt } from "../services/hot-restart.js";
|
||||
import { hasRemoteTerminationReceipt, remoteTerminationReceipt, resolveRemoteTerminationLease } from "../services/remote-execution-termination.js";
|
||||
import { createReusableLeaseHandoff } from "../services/environment-lease-handoff.js";
|
||||
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
|
||||
import { announcePluginManifestChange } from "../services/plugin-manifest-revision.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_HISTORY_TEST_DATABASE_URL");
|
||||
describe("durable sandbox creation through the runtime entry point", () => {
|
||||
@@ -37,6 +38,7 @@ describe("durable sandbox creation through the runtime entry point", () => {
|
||||
}, 90_000);
|
||||
beforeEach(async () => {
|
||||
await db.execute(sql`truncate sandbox_creation_operations, companies, environments, plugins cascade`);
|
||||
announcePluginManifestChange();
|
||||
companyId = randomUUID(); environmentId = randomUUID(); pluginId = randomUUID(); runId = randomUUID();
|
||||
const agentId = randomUUID(); creates = 0; destroys = 0; loseReply = false; failDestroy = false; pluginLogin = false; generatedLoginId = false; foreignReceipt = false; beforeReply = undefined; resources.clear();
|
||||
spool = await mkdtemp(join(tmpdir(), "empty-sandbox-web-"));
|
||||
@@ -451,6 +453,7 @@ describe("durable sandbox creation through the runtime entry point", () => {
|
||||
await secrets.createBinding({ companyId, secretId: secret.id, targetType: "environment", targetId: environmentId, configPath: "apiKey" });
|
||||
await db.update(plugins).set({ manifestJson: sql`jsonb_set(${plugins.manifestJson}, '{environmentDrivers,0,configSchema}',
|
||||
'{"type":"object","properties":{"apiKey":{"type":"string","format":"secret-ref"}}}'::jsonb)` }).where(eq(plugins.id, pluginId));
|
||||
announcePluginManifestChange();
|
||||
await db.update(environments).set({ config: { provider: "creation-fixture", image: "fixture", reuseLease: false, apiKey: secret.id } })
|
||||
.where(eq(environments.id, environmentId));
|
||||
loseReply = true;
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { eq, sql, type SQL } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { createDb, plugins, type Db } from "@paperclipai/db";
|
||||
import { createFullDatabaseResponseCapture, installDatabaseResponseCapture } from "./helpers/full-database-response-capture.js";
|
||||
import type { PaperclipPluginManifestV1, PluginEnvironmentDriverDeclaration } from "@paperclipai/shared";
|
||||
import { resolvePluginSandboxProviderConfigSchema, resolvePluginSandboxProviderDriverByKey } from "../services/plugin-environment-driver.js";
|
||||
import {
|
||||
PROVIDER_SCHEMA_REUSE_MS, resolvePluginSandboxProviderConfigSchema, resolvePluginSandboxProviderDriverByKey,
|
||||
} from "../services/plugin-environment-driver.js";
|
||||
import { resolveEnvironmentDriverConfigForRuntime, resolveSandboxProviderSecretRefPaths } from "../services/environment-config.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
@@ -37,6 +39,7 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
return observed;
|
||||
}
|
||||
const lookup = (client: Db, driverKey = key) => resolvePluginSandboxProviderConfigSchema({ db: client, driverKey });
|
||||
const expire = () => vi.setSystemTime(Date.now() + PROVIDER_SCHEMA_REUSE_MS);
|
||||
async function measure<T>(client: Db, operation: () => Promise<T>) {
|
||||
const events: Array<Record<string, unknown>> = [], now = Date.now();
|
||||
const capture = createFullDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event), () => now);
|
||||
@@ -54,8 +57,8 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("sandbox-schema-reads-");
|
||||
db = createDb(temporary.connectionString);
|
||||
}, 120_000);
|
||||
beforeEach(async () => { await db.delete(plugins); });
|
||||
afterEach(async () => { await Promise.all(clients.splice(0).map(client => client.$client.end())); });
|
||||
beforeEach(async () => { vi.useFakeTimers({ toFake: ["Date"] }); await db.delete(plugins); });
|
||||
afterEach(async () => { vi.useRealTimers(); await Promise.all(clients.splice(0).map(client => client.$client.end())); });
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
it("keeps runtime secret paths current without retransferring an unchanged schema", async () => {
|
||||
@@ -65,15 +68,20 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
const id = await seed([driver(key, schema)]), observed = observer();
|
||||
const before = await db.select().from(plugins).where(eq(plugins.id, id));
|
||||
const cold = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key));
|
||||
const reused = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key));
|
||||
expire();
|
||||
const warm = await measure(observed, () => resolveSandboxProviderSecretRefPaths(observed, key));
|
||||
expect(cold.value).toEqual(new Set(["credentials.token"]));
|
||||
expect(reused.value).toEqual(cold.value);
|
||||
expect(warm.value).toEqual(cold.value);
|
||||
expect(cold.calls).toBe(1);
|
||||
expect(reused.calls).toBe(0);
|
||||
expect(warm.calls).toBe(1);
|
||||
expect(cold.bytes).toBeGreaterThan(32 * 1024);
|
||||
expect(warm.bytes).toBeLessThan(256);
|
||||
const changed = { type: "object", allOf: [{ properties: { apiKey: { type: "string", format: "secret-ref" } } }] };
|
||||
await db.update(plugins).set({ manifestJson: manifest(id, [driver(key, changed)]) }).where(eq(plugins.id, id));
|
||||
expire();
|
||||
expect(await resolveSandboxProviderSecretRefPaths(observed, key)).toEqual(new Set(["apiKey"]));
|
||||
const environment = { id: randomUUID(), driver: "sandbox" as const, config: { provider: key, apiKey: randomUUID() } };
|
||||
await expect(resolveEnvironmentDriverConfigForRuntime(observed, null, environment)).rejects.toThrow("companyId");
|
||||
@@ -90,10 +98,13 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
expect(full?.plugin.status).toBe("disabled");
|
||||
expect(await resolvePluginSandboxProviderDriverByKey({ db: observed, driverKey: key, requireRunning: true })).toBeNull();
|
||||
await db.update(plugins).set({ installOrder: 0 }).where(eq(plugins.id, b));
|
||||
expire();
|
||||
expect((await lookup(observed))).toEqual({ title: "other plugin" });
|
||||
await db.delete(plugins).where(eq(plugins.id, b));
|
||||
expire();
|
||||
expect((await lookup(observed))).toEqual({ title: "first" });
|
||||
await db.update(plugins).set({ manifestJson: manifest(a, [driver("replacement")]) }).where(eq(plugins.id, a));
|
||||
expire();
|
||||
expect(await lookup(observed)).toBeNull();
|
||||
});
|
||||
|
||||
@@ -112,6 +123,7 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
const first = await lookup(a);
|
||||
first!.properties = {};
|
||||
expect(await resolveSandboxProviderSecretRefPaths(a, key)).toEqual(new Set(["token"]));
|
||||
expire();
|
||||
const warm = await measure(a, () => lookup(a)), independent = await measure(b, () => lookup(b));
|
||||
expect(warm.bytes).toBeLessThan(256);
|
||||
expect(independent.bytes).toBeGreaterThan(4096);
|
||||
@@ -125,6 +137,7 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
await seed(declarations);
|
||||
const observed = observer();
|
||||
for (let index = 0; index < 33; index++) await lookup(observed, `provider-${index}`);
|
||||
expire();
|
||||
expect((await measure(observed, () => lookup(observed, "provider-32"))).bytes).toBeLessThan(256);
|
||||
expect((await measure(observed, () => lookup(observed, "provider-0"))).bytes).toBeGreaterThan(2048);
|
||||
await lookup(observed, "oversized");
|
||||
@@ -141,6 +154,7 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
return observed.execute(query);
|
||||
} } as unknown as Db;
|
||||
expect((await lookup(guarded, unusual))).toEqual({ title: "exact provider" });
|
||||
expire();
|
||||
fail = true;
|
||||
await expect(lookup(guarded, unusual)).rejects.toThrow("fixture database unavailable");
|
||||
fail = false;
|
||||
@@ -159,6 +173,7 @@ describe.skipIf(!support.supported)("sandbox provider schema reads", () => {
|
||||
return rows;
|
||||
} } as unknown as Db;
|
||||
expect((await lookup(raced))).toEqual({ title: "old" });
|
||||
expire();
|
||||
pause = true;
|
||||
const oldRead = lookup(raced);
|
||||
await paused;
|
||||
|
||||
@@ -29,6 +29,7 @@ import {
|
||||
readConfigValueAtPath,
|
||||
writeConfigValueAtPath,
|
||||
} from "./json-schema-secret-refs.js";
|
||||
import { pluginManifestRevision } from "./plugin-manifest-revision.js";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
|
||||
@@ -229,10 +230,15 @@ export async function resolvePluginEnvironmentDriverByKey(input: {
|
||||
});
|
||||
}
|
||||
|
||||
type CachedProviderSchema = { pluginId: string; revision: string; schema: Record<string, unknown> | null };
|
||||
type CachedProviderSchema = {
|
||||
pluginId: string; revision: string; schema: Record<string, unknown> | null;
|
||||
checkedAt: number; manifestRevision: number; lookup: number;
|
||||
};
|
||||
const providerSchemas = new WeakMap<Db, Map<string, CachedProviderSchema>>();
|
||||
const MAX_CACHED_PROVIDER_SCHEMAS = 32;
|
||||
const MAX_CACHED_PROVIDER_SCHEMA_BYTES = 64 * 1024;
|
||||
export const PROVIDER_SCHEMA_REUSE_MS = 5_000;
|
||||
let providerSchemaLookups = 0;
|
||||
|
||||
export async function resolvePluginSandboxProviderConfigSchema(input: {
|
||||
db: Db; driverKey: string;
|
||||
@@ -240,6 +246,13 @@ export async function resolvePluginSandboxProviderConfigSchema(input: {
|
||||
const { db, driverKey } = input;
|
||||
let cache = providerSchemas.get(db);
|
||||
const cached = cache?.get(driverKey);
|
||||
const manifestRevision = pluginManifestRevision();
|
||||
const startedAt = Date.now();
|
||||
const lookup = ++providerSchemaLookups;
|
||||
if (cached && cached.manifestRevision === manifestRevision
|
||||
&& startedAt >= cached.checkedAt && startedAt - cached.checkedAt < PROVIDER_SCHEMA_REUSE_MS) {
|
||||
return structuredClone(cached.schema);
|
||||
}
|
||||
const unchanged = sql`coalesce(plugin_id = ${cached?.pluginId ?? null}::uuid
|
||||
and revision = ${cached?.revision ?? null}, false)`;
|
||||
const [row] = await db.execute<{
|
||||
@@ -272,15 +285,20 @@ export async function resolvePluginSandboxProviderConfigSchema(input: {
|
||||
if (!cached || cached.pluginId !== row.pluginId || cached.revision !== row.revision) {
|
||||
throw new Error("Sandbox provider schema revision did not match the retained definition");
|
||||
}
|
||||
if (cache!.get(driverKey) === cached) cache!.set(driverKey, { ...cached, checkedAt: startedAt, manifestRevision, lookup });
|
||||
return structuredClone(cached.schema);
|
||||
}
|
||||
const schema = row.configSchema && typeof row.configSchema === "object" && !Array.isArray(row.configSchema)
|
||||
? row.configSchema as Record<string, unknown> : null;
|
||||
const current = cache?.get(driverKey);
|
||||
if (current && current.lookup > lookup) return schema;
|
||||
cache?.delete(driverKey);
|
||||
if (Buffer.byteLength(JSON.stringify(schema)) <= MAX_CACHED_PROVIDER_SCHEMA_BYTES) {
|
||||
if (!cache) { cache = new Map(); providerSchemas.set(db, cache); }
|
||||
if (cache.size >= MAX_CACHED_PROVIDER_SCHEMAS) cache.delete(cache.keys().next().value!);
|
||||
cache.set(driverKey, { pluginId: row.pluginId, revision: row.revision, schema: structuredClone(schema) });
|
||||
cache.set(driverKey, {
|
||||
pluginId: row.pluginId, revision: row.revision, schema: structuredClone(schema), checkedAt: startedAt, manifestRevision, lookup,
|
||||
});
|
||||
}
|
||||
return schema;
|
||||
}
|
||||
|
||||
@@ -45,6 +45,7 @@ import { logger } from "../middleware/logger.js";
|
||||
import { pluginManifestValidator } from "./plugin-manifest-validator.js";
|
||||
import { pluginCapabilityValidator } from "./plugin-capability-validator.js";
|
||||
import { pluginRegistryService } from "./plugin-registry.js";
|
||||
import { announcePluginManifestChange } from "./plugin-manifest-revision.js";
|
||||
import type { PluginWorkerManager, WorkerStartOptions, WorkerToHostHandlers } from "./plugin-worker-manager.js";
|
||||
import type { PluginEventBus } from "./plugin-event-bus.js";
|
||||
import type { PluginJobScheduler } from "./plugin-job-scheduler.js";
|
||||
@@ -1855,7 +1856,7 @@ export function pluginLoader(
|
||||
{ persistFailure: false },
|
||||
);
|
||||
}
|
||||
});
|
||||
}).finally(announcePluginManifestChange);
|
||||
|
||||
log.info(
|
||||
{
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
let revision = 0;
|
||||
|
||||
export function pluginManifestRevision(): number {
|
||||
return revision;
|
||||
}
|
||||
|
||||
export function announcePluginManifestChange(): void {
|
||||
revision += 1;
|
||||
}
|
||||
@@ -33,6 +33,7 @@ import type {
|
||||
import { conflict, notFound } from "../errors.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { TERMINAL_WORKSPACE_PLUGIN_HINT } from "./terminal-workspace-work.js";
|
||||
import { announcePluginManifestChange } from "./plugin-manifest-revision.js";
|
||||
|
||||
function sameBundledPluginRegistration(existing: typeof plugins.$inferSelect, packageName: string, manifest: PaperclipPluginManifestV1): boolean {
|
||||
return existing.pluginKey === manifest.id && existing.packageName === packageName && existing.version === manifest.version && isDeepStrictEqual(existing.manifestJson, manifest);
|
||||
@@ -180,7 +181,8 @@ export function pluginRegistryService(db: Db) {
|
||||
})
|
||||
.where(eq(plugins.id, existing.id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
.then((rows) => rows[0] ?? null)
|
||||
.finally(announcePluginManifestChange);
|
||||
}
|
||||
|
||||
const installOrder = await nextInstallOrder();
|
||||
@@ -202,6 +204,7 @@ export function pluginRegistryService(db: Db) {
|
||||
const rows = await (options.idempotent
|
||||
? insertion.onConflictDoNothing({ target: plugins.pluginKey })
|
||||
: insertion).returning();
|
||||
announcePluginManifestChange();
|
||||
if (rows[0]) return rows[0];
|
||||
const concurrent = await getByKey(manifest.id);
|
||||
if (concurrent && concurrent.status !== "uninstalled" && sameBundledPluginRegistration(concurrent, input.packageName, manifest)) return concurrent;
|
||||
@@ -256,7 +259,7 @@ export function pluginRegistryService(db: Db) {
|
||||
.set(setClause)
|
||||
.where(eq(plugins.id, id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null));
|
||||
.then((rows) => rows[0] ?? null)).finally(announcePluginManifestChange);
|
||||
},
|
||||
|
||||
// ----- Status ---------------------------------------------------------
|
||||
@@ -334,7 +337,7 @@ export function pluginRegistryService(db: Db) {
|
||||
.where(eq(plugins.id, id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
});
|
||||
}).finally(announcePluginManifestChange);
|
||||
},
|
||||
|
||||
// ----- Config ---------------------------------------------------------
|
||||
|
||||
Reference in new issue
Block a user