mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent keeps its own files (AGENT_HOME). A warm native session saves these files at the end of each turn (an agent-file checkpoint) > - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on the small file-storage pool, also when the run changed no file > - A checkpoint that changed files also uploaded the new bytes to object storage while it held that lock. Instruction edits and other runs of the same agent waited for the upload > - With many agents, these long transactions use the 2-connection file-storage pool and slow every agent > - This pull request removes the lock from an unchanged checkpoint and moves the uploads of a changed checkpoint before the lock > - The benefit is that an unchanged checkpoint takes no row lock, and the row lock is no longer held across object-storage I/O ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm agent-file checkpoint: `checkpoint()` in `server/src/services/agent-directory-working-copies.ts`, `agentFileStore.locked` and `apply` in `agent-file-store.ts`, and `persistDurableAgentFiles` in `agent-file-durability.ts`. **Current behavior** - When the run changed no file, the checkpoint still ran `store.locked(..., write = true)`. That transaction locks the agent row (`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the local projection, walks the canonical tree, and compares the durable head. Its only result is the authorization check. - When the run changed files, `persistDurableAgentFiles` uploaded each new file to object storage inside the transaction that holds the agent row lock. **Proposed behavior** - An unchanged checkpoint (`saveUnchanged`) first reads, without a lock, whether the locked file-tree step would change anything (`agentFileProjectionCurrent`): the durable head exists and the local projection is marked at its generation, or durable files are off, no head exists and the local adoption is recorded. If so, it runs only the write authorization (`authorizeSave`): the agent interaction policy and `authorizeInstructionCommit`, in a short transaction without row locks that always rolls back, so it writes nothing. If not, it runs the locked save as on `origin/master`, which adopts, restores or heals the durable head. A de-authorized agent still gets the same error (the existing test "rechecks authorization even for an unchanged warm checkpoint" still passes). - A changed checkpoint runs the same read-only authorization (it consumes no accepted change confirmation), then uploads the run's changed files before it takes the lock (`uploadAgentFilesBeforeCommit`). A run that lost its save permission uploads nothing. The locked commit runs the authorization again, and only that commit consumes a confirmation. The locked commit uses an uploaded object only for a file with the same SHA-256 and size. It uploads any other file itself, as before. **Reason and benefit** The target is 1,000 agents in one controller process. Most warm turns change no agent file. These turns now take no row lock and send fewer statements. A changed turn holds the row lock for less time, because object-storage latency is outside the lock. ## What Changed - `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)` and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a checkpoint, then uploads the run's changed files before `locked`, and passes the uploaded objects to `persistDurableAgentFiles`. `locked` takes an optional map of uploaded objects. - `agent-file-durability.ts`: move the verified upload stream into `uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the durable head without a lock, skips files the head already has at the same path and digest, and uploads the others. If an upload fails, it stops and the locked commit uploads the rest, so errors are the same as before. `persistDurableAgentFiles` uses an uploaded object when the canonical file has the same digest and size. - `agent-directory-working-copies.ts`: an unchanged checkpoint calls `saveUnchanged` instead of `store.locked(..., true)`. - `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head read and two file checks, no lock). - The commit is still inside the agent row lock. The head cannot change between the read and the write of the commit, so it needs no separate compare-and-set. The upload before the lock does not need one either: an object is immutable and verified by digest, and the commit records it only for a file with that digest. ## Verification - New `server/src/__tests__/agent-file-checkpoint-locking.test.ts` (embedded PostgreSQL, object-storage fixture). All tests fail on the code before the change and pass with it: - An unchanged checkpoint completes while another transaction holds the agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR SHARE` statement and only one unlocked read of `agent_file_roots`. - Durable files on, the agent has local files and no head: an unchanged checkpoint creates the head, as on `origin/master` (added after review; failed without the fallback). - The local projection is not marked at the head: an unchanged checkpoint takes the locked path and marks it again (added after review; failed without the fallback). - A changed checkpoint uploads its file while the agent row is not locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from another connection during each upload). The head then records that object. - A run loses its save permission before a changed checkpoint. The checkpoint is refused and uploads nothing (added after review; I saw it fail without the authorization before the upload). - An agent with `agents:suggest-changes` and one accepted change confirmation saves a changed checkpoint. An unchanged checkpoint first leaves the confirmation unconsumed. The changed save succeeds once and consumes the confirmation once; the next change is refused before it uploads (added after review; I saw it fail when the preview committed). - Two warm runs of the same agent save at the same time. Both files are in the canonical tree and in the head, and every head object has the correct digest. - An instruction edit commits while a checkpoint waits in its upload, without a wait for the checkpoint. Then the checkpoint commits. Both changes are in the head. - A save stops between upload and commit (the instruction entry changes while the bytes upload, so the commit refuses). The head does not change and stays restorable. The next save commits the same file. The unused object is never authoritative; the existing object collector removes it after its grace period. - Existing `agent-directory-working-copies.test.ts`, `agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts` pass. - Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`, test "measures checkpoint statements and agent row lock time"): one warm copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that each change one file, with 250 ms object-store latency. Wall times are noisy (load average about 40). Statement counts are exact. | | Before (`origin/master`) | After (final head) | | --- | --- | --- | | Unchanged checkpoint: statements | 26 | 25 (one unlocked head read, and the read-only authorization) | | Unchanged checkpoint: row-lock statements | 2 | 0 | | Unchanged checkpoint: median time | 214 ms | 26–126 ms | | Changed checkpoint: statements | 29 | 47 (the read-only authorization before the upload and one head read) | | Changed checkpoint: median time the agent row is locked | 924 ms | 51–227 ms | | Changed checkpoint: uploads while the agent row is locked | 1 | 0 | The "after" ranges come from two runs at different machine load (load average between about 18 and 40). ## Risks - Master side effects of the locked unchanged save, and what this change does with each: - Kept: the write authorization (`assertAgentInteraction`, `authorizeInstructionCommit`), so a run that lost its permission still gets the same error. - Kept through the fallback: one-time adoption (no head yet), restore of a projection that is not marked at the head, re-mark after a failed persist (a write path removes the marker before it changes files), and heal of a head that lags the root after such a failure. `agentFileProjectionCurrent` returns false in each of these cases, so the locked save runs. - Dropped when the projection is current: the re-mark at the same generation (no change), and the compare of the local tree with the head. With the marker at the head generation, every writer that changed the tree went through the locked save and moved the marker, so the compare finds no difference. An edit made outside Paperclip directly in the canonical directory is not picked up by an unchanged checkpoint any more; the next save that goes through the locked path still records it. - Dropped when the projection is current: the storage-limit check of the canonical tree in `persistDurableAgentFiles`. An over-quota agent whose run changed nothing no longer gets an error from that check; any save that changes files still enforces the limits. - Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction` takes on the actor and target agents (`lock: "update"`). Those locks hold permission changes off a write until it commits. The read-only authorization writes nothing, so there is nothing to hold them off. The locked commit of a changed save still takes them. An unchanged save has no mention texts, so master locked no other agents for it. - Effects of `authorizeSave` outside the rolled-back transaction: none. I read `assertAgentInteraction` (`agentInteractionDenial`: reads only), `authorizeInstructionCommit`, `resolveInstructionActor` (reads only), `authorizationService.decide` (reads only; its memo maps live on the bound actor and on the service instance, both created for this call and dropped after it), and `changeConsentGateService.assertConsented` (one `UPDATE` of the confirmation row, rolled back). None of them publishes live events, queues notifications or wakes, writes activity through another connection, makes external calls, or changes an in-process cache that outlives the call. - Behavior change: on `origin/master`, an unchanged checkpoint of a consent-gated agent (`agents:suggest-changes`) consumed an accepted change confirmation, because the locked transaction committed. Now the fast path does not; only the commit of a changed save consumes it. When the fast path cannot be used (adoption, restore or heal), the locked save runs as on master and can still consume a confirmation. - Not in this change: the manifest is still stored twice in the receipt (`baseline` and `checkpointManifest`). Servers of the previous version read `receipt.baseline` during a deploy overlap, and the next capture needs the stamps in `checkpointManifest`, so neither can be removed safely now. - Not in this change: on the volume-free server, `serial()` still opens a dedicated connection with an advisory-lock transaction for each checkpoint. That lock serializes controllers and is not a row lock; changing it is a separate design decision. - A changed checkpoint can now leave an unused object in storage when its commit is refused after the upload (for example, the instruction entry changes during the upload) or the process stops before the commit. This was already possible for a failed commit. Such an object is never in the durable head, so no reader can reach it, and the existing `collectAgentFileObjects` maintenance removes it after `AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the save stops the save before any upload. - Old and new servers can run at the same time: the head format and the receipt format do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
8f105e743a
commit
a48eaaef57
4 files changed
+495
-26
No files matched your search
@@ -0,0 +1,359 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { Readable } from "node:stream";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { agentFileRoots, agents, authUsers, closeFileStorageDb, companies, companyMemberships, createDb, fileStorageDb, heartbeatRuns,
|
||||
issueThreadInteractions, issues, principalPermissionGrants } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { agentFileStore, fileHash } from "../services/agent-file-store.js";
|
||||
import { agentInstructionWorkingCopyService } from "../services/agent-instruction-working-copies.js";
|
||||
import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js";
|
||||
|
||||
/** Object storage fixture. Each upload records whether the agent row was locked while it ran. */
|
||||
const objects = vi.hoisted(() => ({
|
||||
bytes: new Map<string, Buffer>(),
|
||||
uploads: [] as Array<{ objectKey: string; sha256: string; agentRowLocked: boolean | null }>,
|
||||
probe: null as null | (() => Promise<boolean>),
|
||||
gate: null as null | Promise<void>,
|
||||
started: null as null | (() => void),
|
||||
delayMs: 0,
|
||||
}));
|
||||
vi.mock("../storage/index.js", () => ({ getStorageService: () => ({
|
||||
provider: "s3",
|
||||
async putFile(input: { companyId: string; namespace: string; body: Readable; byteSize: number; sha256: string }) {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of input.body) chunks.push(Buffer.from(chunk));
|
||||
const body = Buffer.concat(chunks);
|
||||
expect(body.length).toBe(input.byteSize);
|
||||
expect(fileHash(body)).toBe(input.sha256);
|
||||
const agentRowLocked = objects.probe ? await objects.probe() : null;
|
||||
objects.started?.();
|
||||
if (objects.gate) await objects.gate;
|
||||
if (objects.delayMs) await new Promise(resolve => setTimeout(resolve, objects.delayMs));
|
||||
const objectKey = `${input.companyId}/${input.namespace}/${randomUUID()}`;
|
||||
objects.bytes.set(objectKey, body);
|
||||
objects.uploads.push({ objectKey, sha256: input.sha256, agentRowLocked });
|
||||
return { objectKey };
|
||||
},
|
||||
async getObject(_companyId: string, objectKey: string) {
|
||||
const body = objects.bytes.get(objectKey);
|
||||
if (!body) throw new Error("fixture object missing");
|
||||
return { stream: Readable.from([body]) };
|
||||
},
|
||||
}) }));
|
||||
|
||||
describe("agent file checkpoint locking", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let observer: ReturnType<typeof createDb>;
|
||||
let copies: ReturnType<typeof agentInstructionWorkingCopyService>;
|
||||
let home: string, companyId: string, agentId: string, userId: string, root: string;
|
||||
const entryFile = "AGENTS.md";
|
||||
const board = () => ({ type: "board" as const, userId, source: "session" as const });
|
||||
|
||||
/** True when another transaction holds the agent row lock. */
|
||||
async function agentRowLocked(): Promise<boolean> {
|
||||
try {
|
||||
await observer.transaction(async tx => { await tx.execute(sql`select id from agents where id = ${agentId} for update nowait`); });
|
||||
return false;
|
||||
} catch (error) {
|
||||
const code = (error as { code?: string; cause?: { code?: string } }).code ?? (error as { cause?: { code?: string } }).cause?.code;
|
||||
if (code === "55P03") return true;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
async function warmRun() {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", responsibleUserId: userId });
|
||||
return (await copies.prepare({ companyId, agentId, runId, cwd: home, warm: true }))!;
|
||||
}
|
||||
async function head() {
|
||||
const [row] = await db.select().from(agentFileRoots).where(and(eq(agentFileRoots.companyId, companyId), eq(agentFileRoots.agentId, agentId)));
|
||||
return row!;
|
||||
}
|
||||
/** Every committed file must resolve to stored bytes with its digest. */
|
||||
async function expectHeadRestorable() {
|
||||
for (const entry of (await head()).entries) {
|
||||
if (entry.kind !== "file" || entry.size === 0) continue;
|
||||
const bytes = objects.bytes.get(entry.objectKey!);
|
||||
expect(bytes, entry.path).toBeDefined();
|
||||
expect(fileHash(bytes!)).toBe(entry.sha256);
|
||||
}
|
||||
}
|
||||
/** Statements sent by both pools while the callback runs. */
|
||||
async function recordStatements<T>(fn: () => Promise<T>): Promise<{ result: T; statements: string[] }> {
|
||||
const statements: string[] = [];
|
||||
const clients = [db.$client, fileStorageDb(db).$client] as unknown as Array<{ options: { debug?: unknown } }>;
|
||||
for (const client of clients) client.options.debug = (_connection: number, query: string) => { statements.push(query); };
|
||||
try { return { result: await fn(), statements }; }
|
||||
finally { for (const client of clients) delete client.options.debug; }
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("agent-file-checkpoint-locking-");
|
||||
db = createDb(database.connectionString);
|
||||
observer = createDb(database.connectionString);
|
||||
copies = agentInstructionWorkingCopyService(db);
|
||||
}, 90_000);
|
||||
afterAll(async () => {
|
||||
await observer?.$client.end({ timeout: 1 });
|
||||
if (db) { await closeFileStorageDb(db, 1); await db.$client.end({ timeout: 1 }); }
|
||||
await database?.cleanup();
|
||||
});
|
||||
beforeEach(async () => {
|
||||
home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "agent-file-checkpoint-locking-")));
|
||||
vi.stubEnv("PAPERCLIP_HOME", home);
|
||||
vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true");
|
||||
objects.bytes.clear(); objects.uploads.length = 0; objects.probe = null; objects.gate = null; objects.started = null; objects.delayMs = 0;
|
||||
companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID();
|
||||
root = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "Owner", adapterConfig: {} });
|
||||
await db.insert(companies).values({ id: companyId, name: "Checkpoint locking", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Owner",
|
||||
adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } });
|
||||
await db.insert(companyMemberships).values([
|
||||
{ companyId, principalType: "user", principalId: userId, membershipRole: "operator" },
|
||||
{ companyId, principalType: "agent", principalId: agentId, membershipRole: "member" },
|
||||
]);
|
||||
await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId,
|
||||
permissionKey: "agents:configure", scope: { agentIds: [agentId] } });
|
||||
await fs.mkdir(root, { recursive: true });
|
||||
await fs.writeFile(path.join(root, entryFile), "# Policy\n");
|
||||
});
|
||||
afterEach(async () => {
|
||||
vi.unstubAllEnvs();
|
||||
await fs.rm(home, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("saves an unchanged warm checkpoint without the agent row lock or a head write", async () => {
|
||||
const copy = await warmRun();
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
const generation = (await head()).generation;
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
let locked!: () => void;
|
||||
const lockTaken = new Promise<void>(resolve => { locked = resolve; });
|
||||
const holder = observer.transaction(async tx => {
|
||||
await tx.execute(sql`select id from agents where id = ${agentId} for update`);
|
||||
locked();
|
||||
await held;
|
||||
});
|
||||
await lockTaken;
|
||||
try {
|
||||
const { result, statements } = await recordStatements(() => Promise.race([
|
||||
copies.checkpointWarm({ companyId, runId: copy.runId }),
|
||||
new Promise<never>((_, reject) => setTimeout(() => reject(new Error("checkpoint waited for the agent row lock")), 5_000)),
|
||||
]));
|
||||
expect(result?.state).toBe("warm_saved");
|
||||
expect(result?.receipt?.checkpointState).toBe("unchanged");
|
||||
expect(statements.filter(statement => /for (update|no key update|share)/i.test(statement))).toEqual([]);
|
||||
// One unlocked read of the head generation; no head write.
|
||||
expect(statements.filter(statement => statement.includes("agent_file_roots")).map(statement => statement.trim().split(/\s/)[0]!.toLowerCase())).toEqual(["select"]);
|
||||
} finally {
|
||||
release();
|
||||
await holder;
|
||||
}
|
||||
expect((await head()).generation).toBe(generation);
|
||||
});
|
||||
|
||||
it("adopts the agent's files into a durable head on an unchanged checkpoint when no head exists", async () => {
|
||||
const copy = await warmRun();
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
// Durable files were turned on after this run started: the agent has local files and no head.
|
||||
await db.delete(agentFileRoots).where(and(eq(agentFileRoots.companyId, companyId), eq(agentFileRoots.agentId, agentId)));
|
||||
await fs.rm(path.join(path.dirname(root), "file-sync", "durable-adoption.json"), { force: true });
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
expect((await head())?.entries.map(entry => entry.path)).toContain(entryFile);
|
||||
await expectHeadRestorable();
|
||||
});
|
||||
|
||||
it("takes the locked path for an unchanged checkpoint when the local projection is not marked at the head", async () => {
|
||||
const copy = await warmRun();
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
await fs.rm(path.join(path.dirname(root), "file-sync", "durable-generation"), { force: true });
|
||||
const { result, statements } = await recordStatements(() => copies.checkpointWarm({ companyId, runId: copy.runId }));
|
||||
expect(result?.state).toBe("warm_saved");
|
||||
expect(statements.some(statement => /for update/i.test(statement))).toBe(true);
|
||||
expect(await fs.readFile(path.join(path.dirname(root), "file-sync", "durable-generation"), "utf8")).toBe((await head()).generation);
|
||||
});
|
||||
|
||||
it("uploads changed bytes before it takes the agent row lock, then commits the uploaded object", async () => {
|
||||
const copy = await warmRun();
|
||||
objects.uploads.length = 0;
|
||||
objects.probe = agentRowLocked;
|
||||
await fs.writeFile(path.join(copy.localRoot, "notes.md"), "turn one");
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
expect(objects.uploads).toEqual([{ objectKey: expect.any(String), sha256: fileHash(Buffer.from("turn one")), agentRowLocked: false }]);
|
||||
const entry = (await head()).entries.find(candidate => candidate.path === "notes.md");
|
||||
expect(entry).toMatchObject({ kind: "file", sha256: fileHash(Buffer.from("turn one")), objectKey: objects.uploads[0]!.objectKey });
|
||||
await expectHeadRestorable();
|
||||
});
|
||||
|
||||
it("uploads nothing when the run lost its save permission before the checkpoint", async () => {
|
||||
const copy = await warmRun();
|
||||
const before = await head();
|
||||
objects.uploads.length = 0;
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId));
|
||||
await db.update(companyMemberships).set({ membershipRole: "member" }).where(eq(companyMemberships.companyId, companyId));
|
||||
await fs.writeFile(path.join(copy.localRoot, "refused.md"), "must not upload");
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.errorCode).toBe("AGENT_FILES_CHECKPOINT_UNSTABLE");
|
||||
expect(objects.uploads).toEqual([]);
|
||||
expect((await head()).generation).toBe(before.generation);
|
||||
});
|
||||
|
||||
it("consumes an accepted change confirmation once, in the commit of the changed checkpoint only", async () => {
|
||||
const copy = await warmRun();
|
||||
const sourceRunId = randomUUID(), issueId = randomUUID(), interactionId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId, agentId, status: "succeeded" });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Review instructions" });
|
||||
await db.delete(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId));
|
||||
await db.update(companyMemberships).set({ membershipRole: "member" }).where(eq(companyMemberships.companyId, companyId));
|
||||
await db.insert(principalPermissionGrants).values([
|
||||
{ companyId, principalType: "agent", principalId: agentId, permissionKey: "agents:suggest-changes" },
|
||||
{ companyId, principalType: "user", principalId: userId, permissionKey: "agents:suggest-changes" },
|
||||
]);
|
||||
await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId, kind: "request_confirmation", status: "accepted", sourceRunId, createdByAgentId: agentId,
|
||||
payload: { version: 1, prompt: "Apply the file change?", detailsMarkdown: "```diff\n+approved\n```", target: { type: "custom", key: `agent:${agentId}:instructions`, revisionId: "proposal" } },
|
||||
result: { version: 1, outcome: "accepted" }, resolvedByUserId: userId, resolvedAt: new Date(),
|
||||
});
|
||||
const confirmation = async () => (await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0]!.result;
|
||||
// A turn that changes no file keeps the confirmation for the change it approves.
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))).toMatchObject({ state: "warm_saved", errorCode: null });
|
||||
expect(await confirmation()).not.toHaveProperty("consumedByRunId");
|
||||
objects.uploads.length = 0;
|
||||
await fs.writeFile(path.join(copy.localRoot, "approved.md"), "approved change");
|
||||
const saved = (await copies.checkpointWarm({ companyId, runId: copy.runId }))!;
|
||||
expect(saved).toMatchObject({ state: "warm_saved", errorCode: null });
|
||||
expect(await confirmation()).toMatchObject({ consumedByRunId: copy.runId });
|
||||
expect((await head()).entries.find(entry => entry.path === "approved.md")).toMatchObject({ sha256: fileHash(Buffer.from("approved change")) });
|
||||
expect(objects.uploads).toHaveLength(1);
|
||||
// The confirmation is spent: the next change is refused before it uploads.
|
||||
await fs.writeFile(path.join(copy.localRoot, "unapproved.md"), "second change");
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.errorCode).toBe("AGENT_FILES_CHECKPOINT_UNSTABLE");
|
||||
expect(objects.uploads).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("commits both runs when two warm checkpoints of one agent race", async () => {
|
||||
const first = await warmRun();
|
||||
const second = await warmRun();
|
||||
objects.uploads.length = 0;
|
||||
objects.probe = agentRowLocked;
|
||||
objects.delayMs = 100;
|
||||
await fs.writeFile(path.join(first.localRoot, "first.md"), "from the first run");
|
||||
await fs.writeFile(path.join(second.localRoot, "second.md"), "from the second run");
|
||||
const [a, b] = await Promise.all([
|
||||
copies.checkpointWarm({ companyId, runId: first.runId }),
|
||||
copies.checkpointWarm({ companyId, runId: second.runId }),
|
||||
]);
|
||||
expect([a?.state, b?.state]).toEqual(["warm_saved", "warm_saved"]);
|
||||
expect(objects.uploads.map(upload => upload.agentRowLocked)).toEqual([false, false]);
|
||||
const paths = (await head()).entries.map(entry => entry.path);
|
||||
expect(paths).toEqual(expect.arrayContaining(["first.md", "second.md", entryFile]));
|
||||
expect(await fs.readFile(path.join(root, "first.md"), "utf8")).toBe("from the first run");
|
||||
expect(await fs.readFile(path.join(root, "second.md"), "utf8")).toBe("from the second run");
|
||||
await expectHeadRestorable();
|
||||
});
|
||||
|
||||
it("lets an instruction edit commit while a checkpoint uploads, then keeps both changes", async () => {
|
||||
const copy = await warmRun();
|
||||
objects.uploads.length = 0;
|
||||
let open!: () => void;
|
||||
objects.gate = new Promise<void>(resolve => { open = resolve; });
|
||||
const uploading = new Promise<void>(resolve => { objects.started = resolve; });
|
||||
await fs.writeFile(path.join(copy.localRoot, "memory.md"), "from the run");
|
||||
const checkpoint = copies.checkpointWarm({ companyId, runId: copy.runId });
|
||||
await uploading;
|
||||
// The checkpoint waits inside its upload. The editor must not wait for it.
|
||||
objects.started = null;
|
||||
const editorGate = objects.gate;
|
||||
objects.gate = null;
|
||||
const edited = await Promise.race([
|
||||
agentFileStore(db).write({ companyId, agentId, path: entryFile, bytes: Buffer.from("# Edited policy\n"), baseHash: fileHash(Buffer.from("# Policy\n")) }, board()),
|
||||
new Promise<never>((_, reject) => setTimeout(() => reject(new Error("editor waited for the checkpoint upload")), 10_000)),
|
||||
]);
|
||||
expect(edited.changed).toBe(true);
|
||||
open();
|
||||
await editorGate;
|
||||
expect((await checkpoint)?.state).toBe("warm_saved");
|
||||
expect(await fs.readFile(path.join(root, entryFile), "utf8")).toBe("# Edited policy\n");
|
||||
expect(await fs.readFile(path.join(root, "memory.md"), "utf8")).toBe("from the run");
|
||||
const entries = new Map((await head()).entries.map(entry => [entry.path, entry]));
|
||||
expect(entries.get(entryFile)).toMatchObject({ sha256: fileHash(Buffer.from("# Edited policy\n")) });
|
||||
expect(entries.get("memory.md")).toMatchObject({ sha256: fileHash(Buffer.from("from the run")) });
|
||||
await expectHeadRestorable();
|
||||
});
|
||||
|
||||
it("keeps the committed files when the save stops between upload and commit", async () => {
|
||||
const copy = await warmRun();
|
||||
const before = await head();
|
||||
objects.uploads.length = 0;
|
||||
let open!: () => void;
|
||||
objects.gate = new Promise<void>(resolve => { open = resolve; });
|
||||
await fs.writeFile(path.join(copy.localRoot, "draft.md"), "uploaded but not committed");
|
||||
const uploading = new Promise<void>(resolve => { objects.started = resolve; });
|
||||
const checkpoint = copies.checkpointWarm({ companyId, runId: copy.runId });
|
||||
await Promise.race([uploading, checkpoint]);
|
||||
objects.started = null;
|
||||
// Change the instruction entry while the bytes upload. The commit then refuses the save.
|
||||
await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: "OTHER.md" } })
|
||||
.where(eq(agents.id, agentId));
|
||||
objects.gate = null;
|
||||
open();
|
||||
const failed = (await checkpoint)!;
|
||||
expect(failed.errorCode).toBe("AGENT_FILES_CHECKPOINT_UNSTABLE");
|
||||
expect(objects.uploads.length).toBeGreaterThan(0);
|
||||
const after = await head();
|
||||
expect(after.generation).toBe(before.generation);
|
||||
expect(after.entries.some(entry => entry.path === "draft.md")).toBe(false);
|
||||
await expectHeadRestorable();
|
||||
// The next save after the entry is restored commits the same change.
|
||||
await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, instructionsEntryFile: entryFile } })
|
||||
.where(eq(agents.id, agentId));
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
expect((await head()).entries.find(entry => entry.path === "draft.md")).toMatchObject({ sha256: fileHash(Buffer.from("uploaded but not committed")) });
|
||||
await expectHeadRestorable();
|
||||
});
|
||||
|
||||
// Opt in: PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1 pnpm exec vitest run src/__tests__/agent-file-checkpoint-locking.test.ts -t measures
|
||||
// Wall time shares the host's load; statement counts and lock observations are exact.
|
||||
it.skipIf(process.env.PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK !== "1")("measures checkpoint statements and agent row lock time", async () => {
|
||||
const copy = await warmRun();
|
||||
for (let file = 0; file < 50; file += 1) await fs.writeFile(path.join(copy.localRoot, `file-${file}.md`), `content ${file}`);
|
||||
expect((await copies.checkpointWarm({ companyId, runId: copy.runId }))?.state).toBe("warm_saved");
|
||||
const unchanged: Array<{ ms: number; statements: number; rowLocks: number }> = [];
|
||||
for (let attempt = 0; attempt < 10; attempt += 1) {
|
||||
const started = performance.now();
|
||||
const { statements } = await recordStatements(() => copies.checkpointWarm({ companyId, runId: copy.runId }));
|
||||
unchanged.push({ ms: performance.now() - started, statements: statements.length,
|
||||
rowLocks: statements.filter(statement => /for (update|no key update|share)/i.test(statement)).length });
|
||||
}
|
||||
// One changed file per save, with 250 ms object-store latency. Observe the agent row lock from another connection.
|
||||
objects.delayMs = 250;
|
||||
const changed: Array<{ ms: number; statements: number; lockedMs: number }> = [];
|
||||
for (let attempt = 0; attempt < 5; attempt += 1) {
|
||||
await fs.writeFile(path.join(copy.localRoot, "changed.md"), `changed ${attempt}`);
|
||||
let polling = true, firstLocked = 0, lastLocked = 0;
|
||||
const poller = (async () => {
|
||||
while (polling) {
|
||||
if (await agentRowLocked()) { const now = performance.now(); firstLocked ||= now; lastLocked = now; }
|
||||
await new Promise(resolve => setTimeout(resolve, 5));
|
||||
}
|
||||
})();
|
||||
const started = performance.now();
|
||||
const { statements } = await recordStatements(() => copies.checkpointWarm({ companyId, runId: copy.runId }));
|
||||
const ms = performance.now() - started;
|
||||
polling = false;
|
||||
await poller;
|
||||
changed.push({ ms, statements: statements.length, lockedMs: firstLocked ? lastLocked - firstLocked : 0 });
|
||||
}
|
||||
const median = (values: number[]) => Math.round([...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!);
|
||||
console.info(JSON.stringify({
|
||||
unchanged: { medianMs: median(unchanged.map(value => value.ms)), statements: unchanged[0]!.statements, rowLockStatements: unchanged[0]!.rowLocks },
|
||||
changed: { medianMs: median(changed.map(value => value.ms)), statements: changed[0]!.statements,
|
||||
medianAgentRowLockObservedMs: median(changed.map(value => value.lockedMs)), lockObservedMs: changed.map(value => Math.round(value.lockedMs)) },
|
||||
}));
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -423,9 +423,11 @@ export function agentDirectoryWorkingCopyService(db: Db, get: (companyId: string
|
||||
const snapshot = checkpointSnapshot(captured.manifest);
|
||||
const nextHash = directorySnapshotSha256(snapshot);
|
||||
const changed = nextHash !== row.baseHash;
|
||||
// An unchanged run writes no agent file. Its save skips the row lock
|
||||
// when the durable head is already current for the local projection.
|
||||
const { storageWarning } = changed
|
||||
? await store.apply({ companyId: row.companyId, agentId: row.agentId, sourceDir: path.join(captured.directory, "files"), baseline: baseline(row), checkpoint: captured.manifest }, actor(row))
|
||||
: await store.locked(row.companyId, row.agentId, actor(row), true, async () => ({
|
||||
: await store.saveUnchanged(row.companyId, row.agentId, actor(row)).then(() => ({
|
||||
storageWarning: typeof row.receipt?.storageWarning === "string" ? row.receipt.storageWarning : null,
|
||||
}));
|
||||
row = await patch(row, { state: stopped ? (changed ? "saved" : "unchanged") : "warm_saved", baseHash: nextHash,
|
||||
|
||||
@@ -15,7 +15,7 @@ import { AgentFileLimitError, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENT
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
type Owner = { companyId: string; id: string; createdAt: Date };
|
||||
type Head = typeof agentFileRoots.$inferSelect;
|
||||
const where = (owner: Owner) => and(eq(agentFileRoots.companyId, owner.companyId), eq(agentFileRoots.agentId, owner.id));
|
||||
const where = (owner: Pick<Owner, "companyId" | "id">) => and(eq(agentFileRoots.companyId, owner.companyId), eq(agentFileRoots.agentId, owner.id));
|
||||
const marker = (root: string) => path.join(path.dirname(root), "file-sync", "durable-generation");
|
||||
const adoptionReceipt = (root: string) => path.join(path.dirname(root), "file-sync", "durable-adoption.json");
|
||||
export const durableAgentFilesEnabled = () => process.env.PAPERCLIP_DURABLE_AGENT_FILES === "true";
|
||||
@@ -51,6 +51,28 @@ async function atomicPrivateWrite(filename: string, contents: string) {
|
||||
} finally { await fs.rm(temporary, { force: true }); }
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the locked file-tree step of a save would change nothing for an
|
||||
* unchanged run: the durable head exists and the local projection is marked at
|
||||
* its generation (so restore is a no-op and persist only re-marks it), or
|
||||
* durable files are off, no head exists and the one-time local adoption is
|
||||
* already recorded (so persist returns at once). A false result asks the caller
|
||||
* to take the locked path, which adopts, restores or heals the head.
|
||||
*/
|
||||
export async function agentFileProjectionCurrent(db: Db, owner: { companyId: string; id: string }, root: string): Promise<boolean> {
|
||||
const [head] = await db.select({ generation: agentFileRoots.generation }).from(agentFileRoots).where(where(owner));
|
||||
const exists = (filename: string) => fs.stat(filename).then(() => true, (error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return false;
|
||||
throw error;
|
||||
});
|
||||
if (!head) return !durableAgentFilesEnabled() && await exists(path.join(path.dirname(root), "file-sync", "adopted.json"));
|
||||
const generation = await fs.readFile(marker(root), "utf8").catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null;
|
||||
throw error;
|
||||
});
|
||||
return generation === head.generation && await fs.stat(root).then(info => info.isDirectory(), () => false);
|
||||
}
|
||||
|
||||
/** Under the agent row lock. A durable head remains authoritative even after
|
||||
* the opt-in is removed; silently falling back would lose acknowledged writes. */
|
||||
export async function restoreDurableAgentFiles(tx: Tx, owner: Owner, root: string): Promise<boolean> {
|
||||
@@ -136,9 +158,69 @@ export async function restoreDurableAgentFiles(tx: Tx, owner: Owner, root: strin
|
||||
} finally { await fs.rm(temporary, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
/** Objects uploaded before the agent row lock, by content (`sha256:size`). */
|
||||
export type PreparedAgentFileObjects = ReadonlyMap<string, string>;
|
||||
const contentKey = (sha256: string, size: number) => `${sha256}:${size}`;
|
||||
|
||||
/** Stream one file to object storage. The upload fails unless the bytes match
|
||||
* the expected size and digest, so a stored object always has that content. */
|
||||
async function uploadAgentFile(service: ReturnType<typeof storage>, owner: { companyId: string; id: string },
|
||||
filename: string, size: number, sha256: string, signal: AbortSignal): Promise<string> {
|
||||
const handle = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||
const source = handle.createReadStream();
|
||||
const hash = createHash("sha256");
|
||||
let read = 0;
|
||||
const body = new Transform({ transform(chunk: Buffer, _encoding, callback) {
|
||||
read += chunk.length;
|
||||
if (read > size) return callback(new Error("Agent file changed during upload"));
|
||||
hash.update(chunk); callback(null, chunk);
|
||||
}, flush(callback) {
|
||||
callback(read === size && hash.digest("hex") === sha256 ? null : new Error("Agent file changed during upload"));
|
||||
} });
|
||||
try {
|
||||
const [result] = await Promise.all([service.putFile({ companyId: owner.companyId, namespace: `agent-files/${owner.id}`,
|
||||
originalFilename: null, contentType: "application/octet-stream", body,
|
||||
byteSize: size, sha256, signal }), pipeline(source, body, { signal })]);
|
||||
return result.objectKey;
|
||||
} finally { source.destroy(); body.destroy(); await handle.close(); }
|
||||
}
|
||||
|
||||
/**
|
||||
* Upload the files a run changed before its save takes the agent row lock, so
|
||||
* the locked commit only records object keys. Objects are immutable and
|
||||
* verified by digest; the commit uses one only for a file with the same digest
|
||||
* and size. An object that no commit uses (a refused save, or a crash before
|
||||
* the commit) is never authoritative and is removed by the object collector
|
||||
* after its grace period. This step is an optimization: a file it could not
|
||||
* upload is uploaded by the locked commit as before.
|
||||
*/
|
||||
export async function uploadAgentFilesBeforeCommit(db: Db, owner: { companyId: string; id: string }, sourceDir: string,
|
||||
files: ReadonlyArray<{ path: string; size: number; sha256: string }>): Promise<PreparedAgentFileObjects> {
|
||||
const prepared = new Map<string, string>();
|
||||
if (files.length === 0) return prepared;
|
||||
try {
|
||||
const [head] = await db.select({ entries: agentFileRoots.entries }).from(agentFileRoots).where(where(owner));
|
||||
if (!head && !durableAgentFilesEnabled()) return prepared;
|
||||
const service = storage();
|
||||
const committed = new Map(head?.entries.map(entry => [entry.path, entry]));
|
||||
const signal = AbortSignal.timeout(30_000);
|
||||
for (const file of files) {
|
||||
const key = contentKey(file.sha256, file.size);
|
||||
const old = committed.get(file.path);
|
||||
if (file.size === 0 || prepared.has(key) || (old?.kind === "file" && old.sha256 === file.sha256 && old.size === file.size)) continue;
|
||||
const filename = await assertInstructionPathSafe(sourceDir, instructionPath(file.path));
|
||||
prepared.set(key, await uploadAgentFile(service, owner, filename, file.size, file.sha256, signal));
|
||||
}
|
||||
} catch {
|
||||
// The locked commit uploads whatever is missing and reports its own errors.
|
||||
}
|
||||
return prepared;
|
||||
}
|
||||
|
||||
/** Publish immutable bytes before the database head. A failed/uncertain database
|
||||
* commit never removes uploaded bytes that may have become authoritative. */
|
||||
export async function persistDurableAgentFiles(tx: Tx, owner: Owner, root: string, importingLegacy = false) {
|
||||
export async function persistDurableAgentFiles(tx: Tx, owner: Owner, root: string, importingLegacy = false,
|
||||
prepared: PreparedAgentFileObjects = new Map()) {
|
||||
const [previous] = await tx.select().from(agentFileRoots).where(where(owner));
|
||||
if (!previous && !durableAgentFilesEnabled()) return;
|
||||
const service = storage();
|
||||
@@ -161,26 +243,10 @@ export async function persistDurableAgentFiles(tx: Tx, owner: Owner, root: strin
|
||||
if (entry.kind !== "file") throw new Error("Agent files cannot contain symbolic links");
|
||||
if (entry.size === undefined || entry.mode === undefined || !entry.hash) throw new Error("Agent file metadata is incomplete");
|
||||
const old = before.get(relative);
|
||||
let objectKey = old?.kind === "file" && old.sha256 === entry.hash && old.size === entry.size ? old.objectKey : null;
|
||||
let objectKey = old?.kind === "file" && old.sha256 === entry.hash && old.size === entry.size ? old.objectKey
|
||||
: prepared.get(contentKey(entry.hash, entry.size)) ?? null;
|
||||
if (!objectKey && entry.size !== 0) {
|
||||
const filename = await assertInstructionPathSafe(root, relative);
|
||||
const handle = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||
const source = handle.createReadStream();
|
||||
const hash = createHash("sha256");
|
||||
let size = 0;
|
||||
const body = new Transform({ transform(chunk: Buffer, _encoding, callback) {
|
||||
size += chunk.length;
|
||||
if (size > entry.size!) return callback(new Error("Agent file changed during upload"));
|
||||
hash.update(chunk); callback(null, chunk);
|
||||
}, flush(callback) {
|
||||
callback(size === entry.size && hash.digest("hex") === entry.hash ? null : new Error("Agent file changed during upload"));
|
||||
} });
|
||||
try {
|
||||
const [result] = await Promise.all([service.putFile({ companyId: owner.companyId, namespace: `agent-files/${owner.id}`,
|
||||
originalFilename: null, contentType: "application/octet-stream", body,
|
||||
byteSize: entry.size, sha256: entry.hash, signal }), pipeline(source, body, { signal })]);
|
||||
objectKey = result.objectKey;
|
||||
} finally { source.destroy(); body.destroy(); await handle.close(); }
|
||||
objectKey = await uploadAgentFile(service, owner, await assertInstructionPathSafe(root, relative), entry.size, entry.hash, signal);
|
||||
}
|
||||
entries.push({ path: relative, kind: "file", mode: entry.mode, size: entry.size, sha256: entry.hash, objectKey });
|
||||
}
|
||||
|
||||
@@ -16,7 +16,7 @@ import { authorizeInstructionCommit, authorizeInstructionRead } from "./agent-in
|
||||
import { assertInstructionPathSafe, instructionPath, instructionBytes, materializeInstructionBytes, readInstructionBytes, MAX_INSTRUCTION_BYTES } from "./agent-instruction-files.js";
|
||||
import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js";
|
||||
import type { AuthorizationActor } from "./authorization.js";
|
||||
import { durableAgentFilesEnabled, invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles } from "./agent-file-durability.js";
|
||||
import { agentFileProjectionCurrent, durableAgentFilesEnabled, invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles, uploadAgentFilesBeforeCommit, type PreparedAgentFileObjects } from "./agent-file-durability.js";
|
||||
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
type Agent = typeof agents.$inferSelect;
|
||||
@@ -193,7 +193,8 @@ export async function isAgentFileSaveRecorded(tx: Tx, agent: Agent, relative: st
|
||||
export function agentFileStore(db: Db) {
|
||||
db = fileStorageDb(db);
|
||||
async function locked<T>(companyId: string, agentId: string, actor: AuthorizationActor, write: boolean,
|
||||
fn: (tx: Tx, agent: Agent, root: string, bound: AuthorizationActor, onRollback: (restore: () => Promise<void>) => void) => Promise<T>, texts: string[] = []) {
|
||||
fn: (tx: Tx, agent: Agent, root: string, bound: AuthorizationActor, onRollback: (restore: () => Promise<void>) => void) => Promise<T>, texts: string[] = [],
|
||||
prepared?: PreparedAgentFileObjects) {
|
||||
return db.transaction(async tx => {
|
||||
if (write) await assertAgentInteraction(tx, { companyId, actorAgentId: actor.type === "agent" ? actor.agentId : null,
|
||||
operation: "configure", targetAgentId: agentId, lockAgentIds: texts.flatMap(extractAgentMentionIds), lock: "update" });
|
||||
@@ -205,7 +206,7 @@ export function agentFileStore(db: Db) {
|
||||
let restore: (() => Promise<void>) | undefined;
|
||||
try {
|
||||
const result = await fn(tx, agent, root, bound, callback => { restore = callback; });
|
||||
if (write) await persistDurableAgentFiles(tx, agent, root);
|
||||
if (write) await persistDurableAgentFiles(tx, agent, root, false, prepared);
|
||||
return result;
|
||||
} catch (error) {
|
||||
await restore?.();
|
||||
@@ -220,8 +221,42 @@ export function agentFileStore(db: Db) {
|
||||
runId: origin ? origin.runId : actor.runId, responsibleUserId: origin ? null : actor.type === "board" ? actor.userId : actor.onBehalfOfUserId,
|
||||
action: "agent.files_updated", entityType: "agent", entityId: agent.id, details });
|
||||
}
|
||||
/** The write authorization of `locked`, without the row lock or the file
|
||||
* tree, in a transaction that always rolls back. It writes nothing: an
|
||||
* accepted change confirmation stays unconsumed, so only the locked commit
|
||||
* of a save consumes it. It is enough for a save that changes no file, and
|
||||
* it stops a save that will be refused before that save uploads any bytes. */
|
||||
async function authorizeSave(companyId: string, agentId: string, actor: AuthorizationActor) {
|
||||
const rollback = Symbol("authorization preview");
|
||||
await db.transaction(async tx => {
|
||||
await assertAgentInteraction(tx, { companyId, actorAgentId: actor.type === "agent" ? actor.agentId : null,
|
||||
operation: "configure", targetAgentId: agentId });
|
||||
const [agent] = await tx.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)));
|
||||
if (!agent) throw notFound("Agent not found");
|
||||
await authorizeInstructionCommit(tx, actor, agent);
|
||||
throw rollback;
|
||||
}).catch(error => { if (error !== rollback) throw error; });
|
||||
}
|
||||
/**
|
||||
* The save of a run that changed no file. When the locked file-tree step
|
||||
* would change nothing (see `agentFileProjectionCurrent`), only the
|
||||
* read-only authorization runs, without the row lock. Otherwise the locked
|
||||
* write runs as for any save, so it adopts, restores or heals the durable
|
||||
* head exactly as before.
|
||||
*/
|
||||
async function saveUnchanged(companyId: string, agentId: string, actor: AuthorizationActor) {
|
||||
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, companyId), eq(agents.id, agentId)));
|
||||
if (agent && agentInstructionsBundleMode(agent) !== "external"
|
||||
&& await agentFileProjectionCurrent(db, agent, resolveManagedInstructionsRoot(agent))) {
|
||||
await authorizeSave(companyId, agentId, actor);
|
||||
return;
|
||||
}
|
||||
await locked(companyId, agentId, actor, true, async () => undefined);
|
||||
}
|
||||
return {
|
||||
locked,
|
||||
authorizeSave,
|
||||
saveUnchanged,
|
||||
download: async (companyId: string, agentId: string, relative: string, actor: AuthorizationActor) => {
|
||||
const opened: { file: Awaited<ReturnType<typeof openAgentFile>> } = { file: null };
|
||||
try {
|
||||
@@ -324,6 +359,13 @@ export function agentFileStore(db: Db) {
|
||||
if (mentions.length) changedMentions.set(name, { hash: fileHash(bytes!), text: mentions.map(id => `[@agent](agent://${id})`).join(" ") });
|
||||
}
|
||||
}
|
||||
// Upload the run's changed bytes before the row lock, after a preview of
|
||||
// the same write authorization. The commit below runs that authorization,
|
||||
// validates the tree and records only objects with matching digests.
|
||||
if (input.checkpoint) await authorizeSave(input.companyId, input.agentId, actor);
|
||||
const prepared = input.checkpoint ? await uploadAgentFilesBeforeCommit(db, target, input.sourceDir, input.checkpoint.entries
|
||||
.filter(([name, entry]) => entry.kind === "file" && JSON.stringify(incoming.entries.get(name)) !== JSON.stringify(input.baseline.entries.get(name)))
|
||||
.map(([name, entry]) => ({ path: name, size: entry.size!, sha256: entry.hash! }))) : undefined;
|
||||
return locked(input.companyId, input.agentId, actor, true, async (tx, agent, root, bound) => {
|
||||
if (deriveBundleState(agent).entryFile !== requestedEntry) throw conflict("Agent instructions entry changed; retry the file synchronization");
|
||||
const entryPath = deriveBundleState(agent).entryFile;
|
||||
@@ -392,7 +434,7 @@ export function agentFileStore(db: Db) {
|
||||
...(input.checkpoint ? { snapshots: { source: incoming, current } } : {}) });
|
||||
await audit(tx, agent, bound, { sourceRunId: currentPluginInteractionOrigin()?.runId ?? actor.runId, contract: AGENT_FILES_CONTRACT });
|
||||
return { storageWarning: storageWarning(total, finalEntries.size, fullFile) };
|
||||
}, [...changedMentions.values()].map(value => value.text));
|
||||
}, [...changedMentions.values()].map(value => value.text), prepared);
|
||||
},
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user